Annoyance judgment system, apparatus, method, and program

ABSTRACT

An annoyance judgment system includes: a biological signal measurement section configured to measure an electroencephalogram signal of a user; a database retaining a plurality of monosyllabic speech sounds such that, for each speech sound, the speech sound and a reference latency of an electroencephalogram negative component corresponding to the speech sound are retained in association; a presented-speech sound determination section configured to deteremine a monosyllabic speech sound to be presented by referring to the database; and an annoyance judgment section configured to judge annoyance of the output speech sound by comparing a peak latency of a negative component of the electroencephalogram signal in a range from 50 ms to 350 ms from a starting point, the a starting point being a point in time at which the determined speech sound is presented, against the reference latency corresponding to the determined speech sound that is retained in the database.

This is a continuation of International Application No.PCT/JP2011/006435, with an international filing date of Nov. 18, 2011,which claims priority of Japanese Patent Application Nos. 2010-261372,and 2010-261373, both filed on Nov. 24, 2010, the contents of which arehereby incorporated by reference.

BACKGROUND

1. Technical Field

The present application relates to a technique of assessing (judging)whether a user has been able to comfortably listen to a speech sound ornot.

2. Description of the Related Art

In recent years, people suffering from hypacusia caused by aging areincreasing in number due to the aging society. Due to increasedopportunities for listening to loud music for long hours as well asother influences, there is an increasing number of young peoplesuffering from hypacusia. Moreover, due to the downsizing and improvedperformance of hearing aids, users have come to wear hearing aidswithout a psychological barrier. Against this background, there is anincreasing number of users who wear hearing aids in order to improvetheir conversational aural distinction abilities.

A hearing aid compensates for the deteriorated hearing of a user byincreasing the amplitude of signals of specific frequencies, amongvarious frequencies that compose sounds that are difficult for the userto hear. A hearing aid is required to adjust the amount by which itamplifies sounds, in accordance with the level of deterioration in thehearing of the user. Therefore, before beginning use of a hearing aid,“fitting” is required for adjusting the amount of sound amplification inaccordance with the hearing of each user.

Fitting means keeping the output sound pressure (i.e. fluctuations inair pressure that are perceivable as a sound) of each sound frequency atan MCL (most comfortable level: a sound pressure that is feltcomfortable to a user). Thus, appropriate fitting is yet to be attainedunder (1) an insufficient amount of amplification, or (2) an excessiveamount of amplification. For example, under an insufficient amount ofamplification, the user cannot aurally distinguish audios, thus fallingshort of the purpose of wearing a hearing aid. Under an excessive amountof amplification, the user is capable of audio distinction; however,there is a problem in that the user may feel annoyed by the audios,which prevents them from using the hearing aid over a long time.Therefore, a fitting needs to be done in such a manner that neither (1)nor (2) occurs. Especially in the case of (2), sounds which are louderthan necessary will be presented from the hearing aid, thus possiblyhurting the ears of the user.

A first step of fitting is measuring an audiogram. An “audiogram” refersto a measurement of a smallest sound pressure of a pure tone that allowsit to be heard; for example, a diagram in which, for each of a number ofsounds of different frequencies, the smallest sound pressure (decibelvalue) that the user can aurally comprehend is plotted against frequency(e.g., 250 Hz, 500 Hz, 1000 Hz, 2000 Hz, 4000 Hz).

Next, based on a fitting theory, which is a mathematical function forestimating an amount of amplification for each frequency, an amount ofamplification for each frequency is determined from the resultantaudiogram.

However, from an audiogram/fitting theory-based adjustment alone, onecannot know whether an optimum fitting has been realized for improvingthe intelligibility in aural distinction of conversations. Possiblereasons are, for example: an audiogram is not in one-to-onecorrespondence with a conversational aural distinction ability; a personsuffering from hypacusia has a narrow range of sound pressure that isfelt to him or her as an appropriate loudness, which makes adjustmentdifficult; and so on.

Therefore, upon wearing a hearing aid which has been determined andadjusted by the aforementioned method, a hearing aid suitability test isto be carried out (see, for example, Hiroshi HOSOI et al., HOCHOUKITEKIGOKENSA NO SHISHIN 2008, or “2008 Guidelines For Hearing AidSuitability Test”, 2008). There are two mandatory test items in ahearing aid suitability test: (1) measurement of a speech soundintelligibility curve, and (2) measurement of an ambient noise tolerancelevel.

In the measurement of a speech sound intelligibility curve, both whenwearing a hearing aid and when not wearing a hearing aid (naked ear),monosyllabic speech sounds are presented at 55 dB SPL (Sound pressurelevel), 65 dB SPL, 75 dB SPL, and 85 dB SPL; and the speech soundintelligibility at each sound pressure is plotted for comparison. Then,if the intelligibility appears improved when wearing a hearing aidrelative to when not wearing a hearing aid, it is determined assuitable.

As used herein, “speech sound intelligibility” refers to an index as tohow well a monosyllabic speech sound has been aurally comprehended. Aspeech sound intelligibility reflects a level of aural distinctionduring conversations. A “monosyllabic speech sound” means either asingle vowel or a combination of a consonant and a vowel (e.g., “

(a)”/“

(da)”/“

(shi)”).

Speech sound intelligibility is assessed through the following procedure(see, for example, Kazuoki KODERA, “HOCHOKI FITTINGU NO KANGAEKATA (or“Concept of Hearing Aid Fitting”), Shindan To Chiryosha, 1999, p. 166).First, audios in the 67S list (20 speech sounds) proposed by the JapanAudiological Society are reproduced one by one, which a user is allowedto hear. Next, through oral explanation, writing, or other methods, theuser is asked to answer which speech sound he or she has aurallycomprehended the presented speech sound to be. Then, an evaluatormatches the answers against the speech sounds which have been presented,and calculates a correctness rate, which is a rate of speech sounds thathave been correctly aurally comprehended among the total of 20 speechsounds. This correctness rate is the speech sound intelligibility.

Various techniques have been disclosed in the past concerning methods ofspeech sound intelligibility assessment. For example, Japanese Laid-OpenPatent Publication No. 9-038069 discloses a speech sound intelligibilityassessment method which employs a personal computer (PC) toautomatically perform correctness determination. This publicationproposes a method in which monosyllabic audios are presented to a userby using a PC; the user is asked to answer with a mouse or by touching apen to the display; the answers are received as inputs to the PC; andcorrectness determinations as to the presented audios and answer inputsare automatically made. Since answer inputs are received with a mouse ora pen touch, there is no need for the evaluator to distinguish andanalyze the user's answers (which are given by oral explanation orwriting), whereby the trouble of the evaluator is reduced.

Moreover, for example, Japanese Laid-Open Patent Publication No.6-114038 discloses a speech sound intelligibility assessment method inwhich, after audio presentation, possible choices of speech sounds arepresented in the form of text characters. In this publication, choicesare limited to only a small number so that the relevant speech sound canbe found among the small number of characters, whereby the user'strouble of finding the character is reduced.

On the other hand, in the measurement of an ambient noise tolerancelevel, sounds which are read aloud are simultaneously presented withambient noise, and after the sounds which are read aloud are heard, anassessment is made as to whether the ambient noise is tolerable or not(KODERA, et al., supra). Specifically, sounds which are read aloud arepresented at 65 dB SPL, and ambient noise is presented at 55 dB SPL, anda subjective impression as to whether the ambient noise is tolerable ornot is to be reported. As the subjective impression, it is to bereported whether one can endure using a hearing aid when listening tosounds which are read aloud in the presence of noise, or it is difficultto wear a hearing aid in the presence of noise. The former case isdetermined as suitable, whereas the latter case is determined asunsuitable.

SUMMARY

The prior art technique needs further improvement in view of assesmentof a user state concerning annoyance in speech sound listening.

One non-limiting and exemplary embodiment disclosed herein is directedto provide a way to assess a user state concerning annoyance in speechsound listening.

An annoyance judgment system according to an embodiment disclosed hereincomprises: a biological signal measurement section for measuring anelectroencephalogram signal of a user; a speech sound database retaininga plurality of monosyllabic speech sounds such that, for each speechsound, the speech sound and a reference latency of anelectroencephalogram negative component corresponding to the speechsound are retained in association; a presented-speech sounddetermination section configured to determine a monosyllabic speechsound to be presented by referring to the speech sound database; anoutput section configured to present the determined speech sound to theuser; and an annoyance judgment section configured to judge annoyance ofthe output speech sound by comparing a peak latency of a negativecomponent of the electroencephalogram signal in a range from 50 ms to350 ms from a starting point, the starting point being a point in timeat which the speech sound is presented, against the reference latencycorresponding to the determined speech sound that is retained in thespeech sound database.

According to the present disclosure, there is provided a way to assess auser state concerning annoyance in speech sound listening.

These general and specific aspects may be implemented using a system, amethod, and a computer program, and any combination of systems, methods,and computer programs.

Additional benefits and advantages of the disclosed embodiments will beapparent from the specification and Figures. The benefits and/oradvantages may be individually provided by the various embodiments andfeatures of the specification and drawings disclosure, and need not allbe provided in order to obtain one or more of the same.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram showing the experimental procedure of anelectroencephalogram measurement experiment in outline.

FIG. 2 is a flowchart showing a procedure corresponding to one trial.

FIG. 3A is a diagram showing sound pressure levels measured with asound-level meter under different conditions, and FIG. 3B is a diagramshowing amounts of gain adjustment for different frequencies under eachof six conditions.

FIG. 4A is a diagram showing electrode positions according to theInternational 10-20 system, and FIG. 4B is a diagram showing electrodepositioning for wearing an electroencephalograph.

FIG. 5 is a diagram showing results of subjective reporting on annoyancemade by a visual analog scale.

FIG. 6 is a diagram showing waveforms obtained by taking an arithmeticmean of event-related potentials at Pz based on a point of audiopresentation as a starting point, for each of different subjectiveevaluations concerning annoyance.

FIG. 7 shows audio waveforms under the MF condition that were presentedin an experiment.

FIGS. 8A to 8C show exemplary arithmetic mean waveforms for each groupof speech sounds sharing the same consonant.

FIG. 9 is a diagram showing changes in the latency of an N1 componentfor each group of speech sounds sharing the same consonant depending onsound pressure, and changes in the latency of an N1 component for a puretone depending on sound pressure in conventional studies.

FIG. 10 is a diagram showing, as an illustration of consonant intensityof speech sound audios which are presented as stimulations, sums ofroot-mean-square amplitude (up to that point in time) of theirrespective consonant portions under the MF condition.

FIG. 11 is a diagram showing examples of audio waveforms of consonantshaving a weak consonant intensity and those having a strong consonantintensity.

FIG. 12 is a diagram showing correspondence between results of latencycomparison of the N1 component and results of annoyance judgment.

FIG. 13 is a diagram showing a construction and an environment of usefor an annoyance judgment system 100 for speech sound listeningaccording to Embodiment 1.

FIG. 14 is a diagram showing the hardware construction of an annoyancejudgment apparatus 1 for speech sound listening according to Embodiment1.

FIG. 15 is a diagram showing the functional block construction of theannoyance judgment system 100 for speech sound listening according toEmbodiment 1.

FIG. 16 is a diagram showing an example of a speech sound DB 71.

FIG. 17 is a diagram showing an example of accumulated results ofannoyance judgment using a technique of Embodiment 1.

FIG. 18 is diagrams showing results of annoyance judgment using thetechnique of Embodiment 1, and a conventional speech soundintelligibility curve.

FIG. 19 is a flowchart showing a procedure of processing which isperformed in the annoyance judgment system 100 for speech soundlistening.

FIG. 20 is a diagram showing an exemplary assessment result fordifferent monosyllabic words.

FIG. 21 is a diagram showing the functional block construction of anannoyance judgment system 200 for speech sound listening according toEmbodiment 2.

FIG. 22 is diagrams showing examples of accumulated results of annoyancejudgment using a technique of Embodiment 2.

FIG. 23 is a flowchart showing a processing procedure of the annoyancejudgment system 200 for speech sound listening according to Embodiment2.

FIG. 24 is a diagram showing the functional block construction of anannoyance judgment system 300 for speech sound listening according toEmbodiment 3.

FIG. 25 is a flowchart showing a processing procedure of the annoyancejudgment system 300 for speech sound listening according to Embodiment3.

FIG. 26 is a diagram showing the functional block construction of anannoyance judgment system 400 for speech sound listening according toEmbodiment 4.

FIG. 27 is a diagram showing the hardware construction of an annoyancejudgment apparatus 4 for speech sound listening according to Embodiment4.

FIG. 28 is a flowchart showing a processing procedure of the annoyancejudgment system 400 for speech sound listening according to Embodiment4.

DETAILED DESCRIPTION

In the speech sound intelligibility curve measurement of theaforementioned hearing aid suitability test, a suitable state isdetermined based only on speech sound intelligibility, while giving noconsideration as to whether the user felt annoyed when listening to thespeech sound. Therefore, even if annoyance is felt when listening to thespeech sound, if the intelligibility which is obtained when wearing ahearing aid is higher than that obtained when not wearing a hearing aid,the acoustic aiding process is assessed as suitable. On the other hand,in the measurement of an ambient noise tolerance level, an assessment ismade as to whether the ambient noise is tolerable, while making noassessment as to the annoyance in speech sound listening. In theseassessments, even an acoustic aiding process that allows the user tofeel annoyed in speech sound listening, such that hearing fatigue islikely to occur, may have been determined as suitable. Annoyance inspeech sound listening is a burden on a user who wears a hearing aid ona daily basis.

An annoyance judgment system according to an embodiment disclosed hereincomprises: a biological signal measurement section for measuring anelectroencephalogram signal of user; a speech sound database retaining aplurality of monosyllabic speech sounds such that, for each speechsound, the speech sound and a reference latency of anelectroencephalogram negative component corresponding to the speechsound are retained in association; a presented-speech sounddetermination section configured to determine a monosyllabic speechsound to be presented by referring to the speech sound database; anoutput section configured to present the determined speech sound to theuser; and an annoyance judgment section configured to judge annoyance ofthe output speech sound by comparing a peak latency of a negativecomponent of the electroencephalogram signal in a range from 50 ms to350 ms from a starting point, the starting point being a point in timeat which the speech sound is presented, against the reference latencycorresponding to the determined speech sound that is retained in thespeech sound database.

The speech sound database may keep the association between each speechsound and a reference latency of an electroencephalogram negativecomponent corresponding to the speech sound on the basis of a durationor intensity of a consonant contained in the speech sound.

If the peak latency of the negative component is equal to or smallerthan the reference latency, the annoyance judgment section may judgethat the audio signal is annoying to the user, and if the peak latencyof the negative component is greater than the reference latency, theannoyance judgment section may judge that the audio signal is notannoying to the user.

The annoyance judgment system may further comprise an event-relatedpotential processing section configured to take a summation ofevent-related potentials of the electroencephalogram signal according toa predetermined criterion and configured to output a result of summationto the annoyance judgment section, wherein, the presented-speech sounddetermination section may determine two or more speech sounds; theoutput section sequentially may present the determined speech sounds;and among the determined speech sounds, the event-related potentialprocessing section may take a summation of event-related potentials forspeech sounds of a same speech sound type or a same sound pressure, eachevent-related potential being based on a point in time of presenting therespective speech sound as a starting point.

As the peak latency, the annoyance judgment section may adopt: a pointin time at which a negative component of the electroencephalogram signalin a range from 50 ms to 350 ms from a starting point takes a smallestpotential, the starting point being a point in time at which thedetermined speech sound is presented; or a peak latency that isassociated with a template having a highest degree of matching, amongpreviously-provided templates of N1 component waveforms, with theelectroencephalogram signal.

The event-related potential processing section may take a summation ofevent-related potentials with respect to each consonant, or with respectto each group of speech sounds whose differences in reference latency issmaller than a predetermined value.

The annoyance judgment system may further comprise a result accumulatingdatabase configured to accumulate information indicating a result ofannoyance judgment for the speech sound, wherein the result accumulatingdatabase may accumulate information indicating the result of annoyancejudgment for the speech sound with respect to each speech sound, eachconsonant, or each group of speech sounds whose differences in referencelatency is smaller than a predetermined value.

The annoyance judgment system may further comprise an acoustic aidingprocessing section configured to select a type of acoustic aidingprocess for the presented speech sound, and modify data of the speechsounds retained in the speech sound database based on the selectedacoustic aiding process.

The annoyance judgment system may further comprise a switching sectionconfigured to switch between a calibration mode of determining referencelatencies of negative components for the user and an assessment mode ofassessing annoyance, wherein, in the calibration mode, the switchingsection may cause the presented-speech sound determination section toselect a vowel, and calculate a reference latency for each speech soundbased on a latency of the negative component for the vowel; and afterswitching to the assessment mode, the switching section may cause theannoyance judgment section to compare the peak latency of the negativecomponent against the calculated reference latency.

In the calibration mode, when a vowel is selected by thepresented-speech sound determination section, the switching section mayset a latency of an N1 component for the vowel as a reference latencyfor the vowel, and calculate a reference latency for each consonant byadding a positive value which is adapted to a duration or intensity of aconsonant portion to the reference latency for the vowel.

An annoyance judgment apparatus according to another embodimentdisclosed herein comprises: a presented-speech sound determinationsection configured to determine a monosyllabic speech sound to bepresented by referring to a speech sound database retaining a pluralityof monosyllabic speech sounds such that, for each speech sound, thespeech sound and a reference latency of an electroencephalogram negativecomponent corresponding to the speech sound are retained in association;an annoyance judgment section configured to, in an electroencephalogramsignal of a user measured by a biological signal measurement section,compare a peak latency of a negative component of theelectroencephalogram signal in a range from 50 ms to 350 ms from astarting point, the starting point being a point in time at which thespeech sound is presented to the user by an output section, against thereference latency corresponding to the determined speech sound that isretained in the speech sound database, and outputting a differencebetween the peak latency and the reference latency; and an acousticaiding processing section configured to adjust the speech sound based onthe difference output from the annoyance judgment section.

An annoyance judgment method according to another embodiment disclosedherein comprises the steps of: measuring an electroencephalogram signalof a user; determining a monosyllabic speech sound to be presented byreferring to a speech sound database retaining a plurality ofmonosyllabic speech sounds such that, for each speech sound, the speechsound and a reference latency of an electroencephalogram negativecomponent corresponding to the speech sound are retained in association;presenting the determined speech sound to the user; and judgingannoyance of the output speech sound by comparing a peak latency of anegative component of the electroencephalogram signal in a range from 50ms to 350 ms from a starting point, the starting point being a point intime at which the speech sound is presented, against the referencelatency corresponding to the determined speech sound that is retained inthe speech sound database.

A computer program according to another embodiment disclosed herein is acomputer program, stored on a non-transitory computer-readable medium,to be executed by a computer mounted in an annoyance judgment system forspeech sound listening, wherein the computer program causes the computerin the annoyance judgment system to execute the steps of: receiving anelectroencephalogram signal of a user; determining a monosyllabic speechsound to be presented by referring to a speech sound database retaininga plurality of monosyllabic speech sounds such that, for each speechsound, the speech sound and a reference latency of anelectroencephalogram negative component corresponding to the speechsound are retained in association; presenting the determined speechsound to the user; and judging annoyance of the output speech sound bycomparing a peak latency of a negative component of theelectroencephalogram signal in a range from 50 ms to 350 ms from astarting point, the starting point being a point in time at which thespeech sound is presented, against the reference latency correspondingto the determined speech sound that is retained in the speech sounddatabase.

An annoyance judgment system according to still another embodimentdisclosed herein comprises: a biological signal measurement sectionconfigured to measure an electroencephalogram signal of a user; an audioinput section configured to input an audio signal of an utterance by aspecified speaker; an audio analysis section configured to output atrigger upon detecting a timing at which the audio signal is input, andanalyzing a characteristic feature of the audio concerning a durationand an intensity of a consonant portion; a reference latency estimationsection configured to, based on the characteristic feature analyzed bythe audio analysis section, estimate a reference latency of a negativecomponent; and an annoyance judgment section configured to judgeannoyance by comparing a peak latency of a negative component of theelectroencephalogram signal in a range from 50 ms to 350 ms from thetrigger as a starting point against the reference latency estimated bythe reference latency estimation section.

The annoyance judgment system may further comprise a character outputsection configured to output text information indicating a speech soundfor the specified speaker to utter, wherein an audio signal of anutterance by the specified speaker is input to the audio input sectionbased on the text information having been output from the characteroutput section.

The character output section may further output information concerning asound pressure indicating a loudness with which the specified speaker isto utter the monosyllabic speech sound; and an audio signal of anutterance by the specified speaker may be input to the audio inputsection based on the text information and information concerning soundpressure having been output from the character output section.

The annoyance judgment system may further comprise a presented-speechsound determination section configured to determine a speech sound forthe specified speaker to utter by referring to a previously-providedspeech sound list, wherein the character output section outputs textinformation indicating the speech sound determined by thepresented-speech sound determination section.

The reference latency estimation section may estimate the referencelatency of a negative component based on the characteristic featureanalyzed by the audio analysis section and on the speech sound for thespecified speaker to utter that is determined by the presented-speechsound determination section.

The reference latency estimation section may estimate the referencelatency of a negative component by adding a predetermined positive valueto previously-provided base latency, the predetermined positive valuebeing adapted to a consonant duration or consonant intensity of theaudio.

An annoyance judgment method according to still another embodimentdisclosed herein comprises the steps of: inputting an audio signal of anutterance by a specified speaker; outputting a trigger upon detecting atiming at which the audio signal is input, and analyzing characteristicfeature of the audio concerning a duration and an intensity of aconsonant portion; estimating a reference latency of a negativecomponent, based on the characteristic feature analyzed by the analyzingstep; and judging annoyance by comparing a peak latency of a negativecomponent of the electroencephalogram signal in a range from 50 ms to350 ms from the trigger as a starting point against the referencelatency estimated by the estimating step.

Hereinafter, with reference to the attached drawings, embodiments of anannoyance judgment system for speech sound listening according to thepresent disclosure will be described.

An annoyance judgment system for speech sound listening according to thepresent disclosure is used for making an assessment concerning, as auser state when listening to speech sounds, whether a user felt annoyedor not in speech sound listening, by utilizing his or herelectroencephalogram. More specifically, the present system presents amonosyllabic speech sound as an audio, and assesses annoyance in speechsound listening, where an event-related potential of the userelectroencephalogram which is measured based on audio presentation as astarting point is utilized as an index.

Now, the terminology used in the present specification will bedescribed. An “event-related potential (ERP)” means a portion of anelectroencephalogram (EEG), referring to a transient potentialfluctuation in the brain which occurs in temporal relationship with anexternal or internal event. To “present an audio” means to output anauditory stimulation (also referred to as an “audio stimulation”). Forexample, an audio may be output through a loudspeaker. Note that thetype of loudspeaker may be arbitrary. It may be a loudspeaker which isplaced on the floor or on a stand, or may be loudspeakers in the form ofheadphones. However, in order to correctly perform an assessment, anyloudspeaker needs to be able to accurately make an output at adesignated sound pressure. “Assessment” may also be used in the sense of“judgment”.

Through a speech sound intelligibility assessment and through a detailedanalysis of the user state at the time of assessment, the inventors havefound that an assessment of “annoyance”, as a measure of how annoyed auser is, is needed. This will be specifically described below.

In a speech sound intelligibility assessment, an assessment is made asto whether each speech sound was aurally distinguished (◯) or not (X),and the number of speech sounds that have been successfully aurallydistinguished is divided by the number of speech sounds subjected toassessment (i.e., 20 in the case of the 67S list). Therefore, the resultdoes not reflect any user state when listening to speech sounds.

However, in actuality, there may be cases where an aural distinction ismade in comfort as well as cases where an aural distinction is made indiscomfort. A speech sound intelligibility assessment is a short-timeassessment which takes place at a hearing aid shop. The fact as towhether the user feels annoyed or not is irrelevant to the assessment;therefore, unless it is so annoying that it is intolerable, the userwill be willing to carry out the assessment task even if slightlyannoyed.

However, in the case where a hearing aid is worn for long hours on adaily basis, it would be a burden to the user if he or she had totolerate acoustic annoyance for a long time.

In view of these situations, the inventors have arrived at the thoughtthat assessment needs to be made separately with respect to differentuser states when listening to speech sounds: when no “patience forannoyance” is needed; and when some “patience for annoyance” is needed.Since annoyance pertains to a process in the brain during speech soundlistening, there is a possibility that it can be assessed throughelectroencephalogram measurement.

1. EXPERIMENTAL OUTLINE

With a view to realizing annoyance judgment in speech sound listening,the inventors have conducted the following experiment for identifying anelectroencephalogram characteristic component which reflects annoyance.

On the premise of presenting a monosyllabic speech sound in the form ofan audio and asking a user to think of a speech sound corresponding tothe audio, an electroencephalogram measurement experiment was conductedwhere an event-related potential was measured based on audiopresentation as a starting point. In the experiment, it was asked thatsubjective reports on annoyance in speech sound listening be made. Then,based on the subjective reports on annoyance, an arithmetic mean ofevent-related potentials was taken. Note that the step of asking tothink of a speech sound corresponding to an audio is not essential inannoyance judgment.

The inventors have thus found that, in the event-related potential basedon audio presentation as a starting point, a negative component (N1component) at a latency of about 200 ms will have its latency decreasedas the annoyance with respect to the audio increases. They have furtherfound that the latency of the N1 component varies depending ondifferences in characteristic features, e.g., consonant duration andconsonant intensity, from speech sound to speech sound. “Latency”represents, based on the point in time of presenting an audiostimulation as a starting point, an amount of time which lapses before apositive component or negative component peak appears.

From these findings, the inventors have found that annoyance judgment inspeech sound listening can be made based on the latency of a negativecomponent (N1 component). With this technique, as a user state whenlistening to speech sounds, an assessment as to whether the user wasbeing annoyed can be made in an objective and quantitative manner foreach speech sound.

These will be described in more detail below. Firstly, anelectroencephalogram measurement experiment which was conducted by theinventors in order to realize annoyance judgment in speech soundlistening will be described. Thereafter, as an embodiment, an outline ofan annoyance judgment apparatus for speech sound listening whichassesses comfortableness of speech sound listening, as well as aconstruction and operation of an annoyance judgment system for speechsound listening which includes the annoyance judgment apparatus forspeech sound listening, will be described.

2. ELECTROENCEPHALOGRAM MEASUREMENT EXPERIMENT

In the electroencephalogram measurement experiment, a relationshipbetween the subjective reports on annoyance which were acquired afteraudio presentation and an event-related potential based on the audio asa starting point was examined. Hereinafter, with reference to FIG. 1 toFIG. 6, the experimental setting and experimental results of theelectroencephalogram measurement experiment will be described.

Thirteen undergraduate or graduate students with normal hearingparticipated in the experiment.

FIG. 1 shows the experimental procedure of the electroencephalogrammeasurement experiment in outline. First, a monosyllabic audio waspresented in Procedure A. The particulars of the presented audios willbe described later. Next, in Procedure B, each participant was allowedto hear an audio, and asked to write down a hiragana corresponding tothe audio as he or she heard it. The conditions of the presented audioswere kept unvaried, while only the speech sound type was varied.Procedures A and B were repeated five times (5 trials). Then, inProcedure C, the participant was asked to make a subjective evaluationconcerning annoyance and the like with respect to each audio that waspresented in Procedure A. The subjective evaluation was based on avisual analog scale (100-step evaluation), and was made by using a touchpanel. This was repeated 12 blocks, where 1 block consisted of ProcedureA to Procedure C as above (totaling 60 trials). For each block, thesound pressure and distortion conditions of the presented audios werevaried in random order.

FIG. 2 is a flowchart showing a procedure corresponding to one block.

At step S11, a monosyllabic audio is presented to an experimentalparticipant.

At step S12, the participant thinks of a corresponding text characterupon hearing the monosyllabic audio.

At step S13, the participant writes down the text charactercorresponding to the audio as he or she heard it.

At step S14, the number of times that the audios have been presented iscounted. While the number of presentations is equal to or less than 4,the process returns to step S11. When the number of presentationsreaches 5, the process proceeds to step S15, where the number ofpresentations is reset.

At step S15, the participant answers with a subjective perception of theaudio which was heard at step S11.

From among unvoiced consonants which are supposed to induce mistakes inaural comprehension, 8 sounds (

(ki),

(ku),

(shi),

(su),

(ta),

(te),

(to),

(ha)) in the 67S list as proposed by the Japan Audiological Society wereselected as the speech sounds to be presented as stimulations. Speechsounds with adjusted frequency gains were used, thus to controlannoyance for participants with normal hearing. A “frequency gain”refers to a gain (i.e., a circuit gain or rate of amplification) foreach of a number of frequency bands.

For frequency gain adjustment, three sound pressure levels (Large,Middle, Small) X two distortion levels (Flat, Distorted) were employed,totaling six conditions, as are detailed in (1) to (6) below. In thepresent specification, large sound pressure and flat (no distortion)conditions may be referred to as LF condition (an acronym of Large andFlat), for example.

(1) LF (Large Flat) condition: the gain was increased by 20 dB acrossentire frequency band, meant as an audio which had a large soundpressure and was easy to aurally distinguish. (2) LD (Large Distorted)condition: the gain was uniformly increased by 20 dB relative to the MDcondition, meant as an audio which had a large sound pressure but wasdifficult to aurally distinguish. (3) MF (Middle Flat) condition: thefrequency gain was not modified, meant as an audio which had a largesound pressure and was easy to aurally distinguish. (4) MD (MiddleDistorted) condition: from an audio of the LF condition, the gain atfrequencies of 250 Hz to 16 kHz was gradually adjusted (decreased) to−30 dB, meant as an audio which was difficult to aurally distinguish.(5) SF (Small Flat) condition: the gain was decreased by 20 dB acrossthe entire frequency band, meant as an audio which had a small soundpressure but was easy to aurally distinguish. (6) SD (Small Distorted)condition: the gain was uniformly decreased by 20 dB relative to the MDcondition, meant as an audio which had a small sound pressure and wasdifficult to aurally distinguish.

FIG. 3A shows classification of six conditions concerning sound pressureand distortion. FIG. 3B shows amounts of gain adjustment for differentfrequencies. The reason why the frequency gain for the high-frequencyband was decreased is in order to reproduce a typical pattern ofhypacusia of elderly people, i.e., gradual high tone loss. The audiostimulations were presented from a loudspeaker with flat frequencycharacteristics.

Each electroencephalogram was recorded from electrodes placed at the Fz,Cz, Pz, C3, and C4 (International 10-20 system) on the scalp, the rightand left temples, and above and below the right eye, on the basis of theright mastoid. A “mastoid” is a protrusion of the cranium below the hindroot of an ear. FIG. 4A shows electrode positions according to theInternational 10-20 system (10-20 System), whereas FIG. 4B showselectrode positioning as to how electrodes were worn in the presentexperiment. The sampling frequency was 200 Hz, and the time constant was1 second. It was subjected to a 0.05 to 20 Hz digital band-pass filteroff-line. As an event-related potential in response to an audiopresentation, a waveform from −200 ms to 1000 ms was cut out based onthe point of audio presentation as a starting point. Herein, “−200 ms”signifies a point in time which is 200 milliseconds before the point ofaudio presentation.

Hereinafter, distribution of results of subjective evaluation andthreshold value setting will be described.

First, results of subjective evaluation will be described. Based on theresults of subjective evaluation, “annoying/not annoying” was labeledrelative to a threshold value which was determined for each participantbased on a method describe below. Hereinafter, these subjectiveevaluation labels will be treated as the user states when listening tospeech sounds.

FIG. 5 shows results of subjective evaluation of different participantsconcerning annoyance. The horizontal axis represents a subjectiveperception evaluation value (1 to 100 on the visual analog scale),whereas the vertical axis represents a ratio (0 to 1) obtained bydividing the frequency distribution by the total number of trials. Eachgraph of FIG. 5 shows a proportion within all trials.

Each solid line in FIG. 5 shows a distribution of results of subjectiveevaluation, whereas each broken line shows a threshold value at whichsubjective evaluations (“annoying/not annoying”) are split. Theinventors determined the threshold value based on the ordinal ranks ofevaluation results (i.e., 1 to 100 on the visual analog scale) of eachindividual person. Specifically, within each individual, the inventorsdefined those evaluation values whose ordinal rank are within thelargest third as “annoying”, and the rest as “not annoying”, thussetting the threshold value therebetween. Herein, identical evaluationresults were treated as pertaining to the same subjective evaluation.

Hereinafter, as event-related potential results, a result of taking anarithmetic mean based on the “annoying”/“not annoying” criteria, aslabeled based on results of subjective evaluation, will be described.

FIG. 6 shows waveforms obtained by taking an arithmetic mean ofevent-related potentials at the parietal (Pz) based on a point ofpresenting an audio stimulation as a starting point. Specifically, FIG.6 shows waveforms each obtained by taking a total arithmetic meanaccording to the “annoying/not annoying” criterion as labeled with theabove method. An arithmetic mean was taken based on the subjectiveevaluations concerning annoyance for respective blocks, under the sixconditions in the above-described measurement experiment. In FIG. 6, thehorizontal axis represents time in units of ms, whereas the verticalaxis represent potential in units of μV. As is clear from the scalesshown in FIG. 6, the lower direction in the graph corresponds to plus(positive), and the upper direction corresponds to minus (negative). InFIG. 6, a solid line represents a total arithmetic mean waveform in thecase of “annoying”, and a broken line represents a total arithmetic meanwaveform in the case of “not annoying”.

It can be seen from FIG. 6 that a negative component (N1 component)which is induced at a latency of about 200 ms has a shorter latency inthe case of “annoying” (solid line) than in the case of “not annoying”(broken line). The latency of the N1 component of each participant was195 ms in the case of “annoying”, and 240 ms in the case of “notannoying”. As a result of t-testing the latencies, a significantdifference was recognized (p<0.05). Thus, it was concluded that thelatency was shorter in the “annoying” case than in the “not annoying”case. It can be said that the latency of an N1 component based onpresentation of an audio stimulation (hereinafter “audio presentation”)as a starting point reflects annoyance as a subjective perception of theuser, and can be used as an index of annoyance in speech soundlistening.

It is known from conventional studies using a pure tone (tone pip, toneburst) that the latency and amplitude of an N1 component in response toan auditory stimulation change in accordance with the intensity and risetime of the sound stimulation (see Suzuki et al., 1985, CHOSEINOKANHANNO—SONOKISOTO RINSHO—, or “Auditory Brain Stem Response—ItsBasics And Clinical Applications—, pp. 384-385). Specifically, as thestimulation sound increases in intensity, the latency decreases and theamplitude (i.e., an absolute value of the difference N1 component−P2component) increases. Moreover, the amplitude decreases as the rise timeof the stimulation sound increases.

In the present experiment, speech sounds were used as auditorystimulations. FIG. 7 shows exemplary audio waveforms under the MFcondition (8 speech sounds:

(ki),

(ku),

(shi),

(su),

(ta),

(te),

(to),

(ha)) that were used in the present experiment. As can be seen from FIG.7, each audio differs in the duration and intensity of the consonantportion and the intensity and rise time of the vowel portion. Forexample, attention may be paid to the duration of the consonant portion(consonant duration). A consonant duration is a length of time whichelapses after a point of occurrence of a waveform (0 ms) until arelatively large rise emerges in the waveform. More specifically, aconsonant duration is a length of time elapsing until a vowel rises. Theconsonant duration was about 80 ms for the ka-row (

(ki),

(ku)), about 170 ms for the sa-row (

(shi),

(su)), about 40 ms for the ta-row (

(ta),

(te),

(to)), and about 130 ms for the ha-row (

(ha)). Moreover, for example, the intensity of the consonant portion wasstronger in the sa-row than in the ka-, ta-, and ha-rows.Notwithstanding these significant row-to-row differences in the audiowaveform, it can also be seen that the characteristic feature of theoverall waveform is kept alike within each given row.

Note that the ka-row includes speech sounds which begin with “k”:specifically, ka, ki, ku, ke, and ko. The sa-row includes speech soundswhich begin with specifically, sa, shi, su, se, and so. The ta-rowincludes speech sounds which begin with “t”: specifically, ta, ti(chi),tu(tsu), te, and to.

From the knowledge on pure tone stimulations from conventional studies,and the differences in audio waveform between speech sounds shown inFIG. 7, there is a possibility that the latency and amplitude of the N1component for a speech sound may differ from speech sound to speechsound. If a characteristic feature of the presented speech sound audiois the cause of a latency fluctuation in the N1 component, the latencyof the N1 component will fluctuate irrespective of “annoyance”.Therefore, incorrect assessments may be made, e.g., assessing anon-annoying audio to be annoying, or assessing an annoying audio to benot annoying.

Therefore, the inventors have taken an arithmetic mean for each givenrow, within which the audio waveforms share similar characteristicfeatures. By taking an arithmetic mean for each row, a certain number oftimes of arithmetic mean that is needed for the analysis of N1 componentlatency is attained. Then, from the arithmetic mean waveform result ofeach row, the influences that differing characteristic features of audiostimulations exert on the N1 component latency were examined. In orderto clarify the relationship between sound pressure and N1 componentlatency, an arithmetic mean was taken for each sound pressure,irrespective of the presence or absence of distortion.

FIGS. 8A to 8C show exemplary arithmetic mean results for differentrows. FIGS. 8A to 8C show results for the ta-row (

(ta),

(te),

(to)), the ha-row (

), and the sa-row (

(shi),

(su)), respectively. In FIGS. 8A to 8C, the horizontal axis representstime in units of ms, whereas the vertical axis represents potential inunits of μV.

As is clear from the scales shown in FIGS. 8A to 8C, the lower directionin the graph corresponds to plus (positive), and the upper directioncorresponds to minus (negative). In each of FIGS. 8A to 8C, a thicksolid line represents a total arithmetic mean waveform under a Largecondition (85 dB), a broken line represents a total arithmetic meanwaveform under a Middle condition (65 dB), and a thin solid linerepresents a total arithmetic mean waveform under a Small condition (45dB). In FIGS. 8A to 8C, each ◯ symbol indicates an N1 component. As theN1 component, a minimum value in a zone from 0 ms to 500 ms is taken,for example. In order to show temporal correspondence between the audiowaveforms and the electroencephalograms, audio waveforms of speechsounds “

(ta)”, “

(ha)”, and “

(shi)” are indicated in FIGS. 8A to 8C as representatives of therespective rows, with their starting points being aligned. It can beseen from FIGS. 8A to 8C that, in each row, the N1 component latencydecreases as the sound pressure increases. It can also be seen thatunder the Large condition of FIG. 8C, the N1 component has its peakbefore the vowel portion rises.

FIG. 9 shows exemplary changes in the latency of each N1 component withsound pressure, as found in the present experiment. FIG. 9 also showschanges in the latency of an N1 component for a pure tone with soundpressure, as found in a conventional study. FIG. 9 indicates thefollowing characteristic features regarding (1) sound intensity and (2)consonant duration. Regarding (1) sound intensity, it can be seen thatthe N1 component latency decreases in each row as the sound pressureincreases. Regarding (2) consonant duration, it can be seen through acomparison between the result for the ta-row and the result for theha-row that the N1 component latency differs depending on consonantduration.

The aforementioned sound intensity and consonant duration will berespectively discussed below.

First, the latency-decreasing characteristics of the N1 componentassociated with increasing sound pressure of the speech sound audio willbe discussed. According to conventional studies, it is known in the caseof a pure tone that the N1 component latency only has a decrease of 10ms for a similar sound pressure increase of 40 dB. On the other hand,upon using the experimental results of the inventors to analyze thedecrease in N1 component latency that is associated with an increase inthe sound pressure of a speech sound audio, it was found that a decreaseof about 100 ms occurs on average among the sa-row, the ta-row, and theha-row, in response to a sound pressure increase of 40 dB (i.e., from 45dB to 85 dB). This indicates differing latency-decreasingcharacteristics of the N1 component between pure tones and speechsounds. Thus, it can be said that the latency-decreasing characteristicsassociated with an increase in the sound pressure of the speech soundaudio had never been clarified before the experiment conducted by theinventors.

Next, the latency-decreasing characteristics of the N1 componentassociated with consonant duration will be discussed. For example, asshown in FIG. 7, the consonant durations of the ta-row and the ha-roware about 40 ms and about 130 ms, respectively, resulting in adifference of about 90 ms. The latencies for the ta-row and the ha-rowunder the Small condition are 236 ms and 322 ms, thus conserving adifference of about 90 ms. Thus, it is considered that these N1components are induced in response to the rise of the vowel. On theother hand, in the sa-row having a stronger consonant intensity than anyother row, consonant duration exerts a different influence. Theconsonant duration of the sa-row is about 170 ms, which is longer thanthe consonant duration (about 130 ms) of the ha-row; however, the N1component latency is shorter for the sa-row than for the ha-row, underall sound pressure conditions. Moreover, the N1 component latency forthe sa-row under Large condition is 156 ms, which is shorter than theconsonant duration of the sa-row itself.

It can be said from these results that, when the consonant duration islonger than a predetermined time (e.g., about 100 ms) and the consonantintensity is strong, an N1 component will be induced in response to theconsonant.

FIG. 10 shows, as an illustration of consonant intensity of speech soundaudios which are presented as stimulations, sums of root-mean-squareamplitude (up to that point in time) of their respective consonantportions under the MF condition. It can be seen from FIG. 10 that theconsonant portion has a higher intensity in the sa-row than in any otherspeech sound. The “consonant intensity” means an amplitude level of anaudio waveform in a time slot corresponding to its consonant portion.The consonant intensity may be determined from a root-mean-square valueof the amplitude of the consonant portion, for example.

FIG. 11 shows examples of audio waveforms of consonants having a weakconsonant intensity and those having a strong consonant intensity. It isindicated that the consonant intensity is stronger in the ma-row “

(mo)” and the na-row “

(ne)” than in the za-row “

(ji)” and the ra-row “

(ri)”. Other than the sa-row, the speech sounds in the ma-row, thena-row, the ya-row, the wa-row, and the ga-row may be said to have astrong consonant intensity, for example.

Therefore, for a speech sound whose consonant portion has a strongintensity, if no N1 component is induced in response to its consonantportion, but an N1 component is induced only in response to its vowelportion (i.e., the N1 component latency is significantly lagged beyondexpectation), it may be determined that the consonant portion was notheard.

Thus, it has become clear through the electroencephalogram measurementexperiment that there is an electroencephalogram component that reflectsa user's subjective evaluation concerning annoyance in speech soundlistening. Specifically, it has been found that a negative potentialhaving a peak at a latency of about 200 ms reflects annoyance. It hasalso been found that the N1 component latency varies from speech soundto speech sound, because of influences of the differing audio waveforms.

The latency of the aforementioned N1 component at the parietal (Pz) foreach subjective evaluation concerning annoyance (FIG. 6) may be definedas a point in time at which the component takes the smallest potentialin the zone in question. Alternatively, templates may be created fromthe waveforms of typical N1 components, and a peak point of the N1component of one template that has the highest degree of matching withthe measured waveform may be regarded as the latency. Note that the N1component latency is identifiable through a method of making acomparison against a predetermined threshold value concerning latency,or the like. The threshold value and template may be those of a typicaluser as previously stored, or may be generated for each individualperson. In this experiment, arithmetic means were taken from the data of13 participants in order to confirm the fact that components whichreflect the subjective perception concerning annoyance appear in anevent-related potential based on a point of audio presentation as astarting point. However, depending on the method of characteristicamount extraction (e.g., wavelet transformation of the waveform) or themethod of identification (e.g., support vector machine learning),identification of a negative component is possible with no summations oronly a small number of summations.

In the present specification, in order to define a component of anevent-related potential, a point in time after the lapse of apredetermined time since a given point is expressed by referring to a“latency of about 200 ms”, for example. This means possible inclusion ofa range around the specific point of 200 ms in time. Generally speaking,there are 30 to 50 ms of differences (shifts) in event-related potentialwaveform between individuals, according to table 1 on p. 30 of“JISHOUKANRENDENI (ERP) MANYUARU —P300 WO CHUSHINNI—(or “Event-RelatedPotential (ERP) Manual—mainly concerning P300—”), edited by KimitakaKAGA et al., Shinohara Shuppan Shinsha, 1995)”. Therefore, the terms“about X ms” and “near X ms” mean that a breadth of 30 to 50 ms mayexist before or after X ms (e.g., 300 ms±30 ms, 750 ms±50 ms). Moreover,as mentioned above, the N1 component latency varies depending on thecharacteristic feature of the speech sound audio. Therefore, in order toat least account for the varying consonant duration (from 0 ms (vowel)to about 200 ms (consonant)), the aforementioned positive component ispreferably treated as having a broader breadth, e.g., a breadth of about150 ms on each of the earlier side and the later side. Accordingly, inthe present embodiment, a “latency of about 200 ms” is meant to indicatea latency falling within the range from 50 ms to 350 ms.

Thus, the inventors have found through their electroencephalogrammeasurement experiment that, in an event-related potential based on thepoint of audio presentation as a starting point, a negative component ata latency of about 200 ms (N1 component) reflects annoyance. Therefore,based on an event-related potential in response to an audio presentation(audio stimulation) as an index, a subjective evaluation of annoyance inspeech sound listening can be realized.

FIG. 12 shows correspondence between N1 component latency and annoyancejudgment, as compiled by the inventors. If the N1 component latency isshorter than a predetermined threshold value, an “annoying” judgment ismade. If the N1 component latency is longer than the predeterminedthreshold value a “not annoying” judgment is made.

Note that a “negative component” would generally mean any potentialwhich is smaller than 0 μV. However, in the present specification, inorder to distinguish whether the user has felt “annoyed” or not, notonly the presence or absence of a “negative” component, but also thelatency of the negative component being longer or shorter than apredetermined threshold value is considered. Note that, throughout thepresent specification, the case where the latency exactly equals thepredetermined threshold value may be judged as indicating either that“the negative component is present” or that “the negative component isabsent”, depending on the application. Specific examples of thethreshold value will be described later.

Hereinafter, an annoyance judgment system for speech sound listeningaccording to embodiments of the present disclosure will be described.The annoyance judgment system for speech sound listening sequentiallypresents monosyllabic speech sounds in the form of audios, anddetermines annoyance in speech sound listening, by relying on thelatency of a negative component in an event-related potential based onthe point of audio presentation as a starting point. This isunprecedentedly realized by the findings of the inventors.

Embodiment 1

Hereinafter, the annoyance judgment system for speech sound listeningwill be first described in outline. Thereafter, the construction andoperation of an annoyance judgment system for speech sound listeningwhich includes the annoyance judgment apparatus for speech soundlistening will be described.

The annoyance judgment system for speech sound listening of the presentembodiment, presents audios, and an event-related potential is measuredbased on each point of audio presentation as a starting point. Then, anegative component at a latency of about 200 ms is detected, andannoyance in speech sound listening is judged.

In the present embodiment, a probe electrode is placed at the parietal(Pz), and a reference electrode was placed at the right or left mastoid,and an electroencephalogram was measured as a potential differencebetween the probe electrode and the reference electrode. Note that thelevels and polarities of the characteristic components of theevent-related potential may vary depending on the position at which theelectrode for electroencephalogram measurement is attached, and themanner in which the reference electrode and the probe electrode are set.However, based on the following description, those skilled in the artwould be able to detect a characteristic component of the event-relatedpotential and make a speech sound intelligibility assessment by makingappropriate modifications depending on the specific reference electrodeand probe electrode. Any such variant is encompassed within the presentdisclosure.

In the above description of the electroencephalogram measurementexperiment, the relative strength of the frequency gain isexperimentally varied for participants with normal hearing, thussimulating the hearing of a person suffering from hypacusia. However,when conducting an assessment for a person suffering from hypacusia,there is no particular need to present speech sounds that are difficultto aurally distinguish. In the present embodiment, it is assumed thataudios which have been adjusted with an optimum gain for each frequencyare presented, based on a fitting theory from audiograms of peoplesuffering from hypacusia that were measured in advance. Note that, inthe case where an assessment is made while the user is wearing a hearingaid, adjustment for the presented audio is unnecessary.

FIG. 13 shows a construction and an environment of use for an annoyancejudgment system 100 for speech sound listening according to the presentembodiment. The annoyance judgment system 100 for speech sound listeningis exemplified so as to correspond to a system construction ofEmbodiment 1 described later.

The annoyance judgment system 100 for speech sound listening includes anannoyance judgment apparatus 1 for speech sound listening, an audiooutput section 11, and a biological signal measurement section 50. Thebiological signal measurement section 50 is connected to at least twoelectrodes A and B. Electrode A is attached at a mastoid of the user 5,whereas electrode B is attached at the parietal (so-called Pz) on thescalp of the user 5.

The annoyance judgment system 100 for speech sound listening presents amonosyllabic speech sound to the user 5 in the form of an audio at acertain sound pressure, and determines whether the latency of an N1component is shorter than a predetermined threshold value or not, in anelectroencephalogram (event-related potential) from the user 5 which ismeasured based on the point of audio presentation as a starting point.Then, based on the presented audio and the result of distinction as tothe latency of the N1 component, it is judged as to whether the userfelt annoyed in speech sound listening.

An electroencephalogram from the user 5 is acquired by the biologicalsignal measurement section 50 based on a potential difference betweenelectrode A and electrode B. The biological signal measurement section50 sends information corresponding to the potential difference(electroencephalogram signal) to the annoyance judgment apparatus 1 forspeech sound listening in a wireless or wired manner. FIG. 13illustrates an example where the biological signal measurement section50 wirelessly sends this information to the annoyance judgment apparatus1 for speech sound listening.

The annoyance judgment apparatus 1 for speech sound listening performssound pressure control of the audio used for annoyance judgment inspeech sound listening, controls presentation timing of the audio, andpresents an audio via the audio output section 11 (e.g., loudspeakers)to the user 5.

FIG. 14 shows a hardware construction of the annoyance judgmentapparatus 1 for speech sound listening according to the presentembodiment. The annoyance judgment apparatus 1 for speech soundlistening includes a CPU 30, a memory 31, and an audio controller 32.These elements are interconnected via a bus 34 so that data exchangeamong them is possible.

The CPU 30 executes a computer program 35 which is stored in the memory31. A processing procedure as illustrated by a subsequently-describedflowchart is described in the computer program 35. In accordance withthe computer program 35, the annoyance judgment apparatus 1 for speechsound listening performs a process of controlling the entire annoyancejudgment system 100 for speech sound listening, by utilizing a speechsound database (DB) 71 which is also stored in the same memory 31. Thisprocess will be described in detail later.

In accordance with instructions from the CPU 30, the audio controller 32generates an audio to be presented, and outputs the generated audiosignal to the audio output section 11 at a designated sound pressure.

Note that the annoyance judgment apparatus 1 for speech sound listeningmay be implemented as a piece of hardware (e.g., a DSP) consisting of asemiconductor circuit having a computer program incorporated therein.Such a DSP can realize all functions of the aforementioned CPU 30,memory 31, and audio controller 32 on a single integrated circuit.

The aforementioned computer program 35 may be distributed on the marketin the form of a product recorded on a storage medium such as a CD-ROM,or transmitted through telecommunication lines such as the Internet.Upon reading the computer program 35, a device having the hardware shownin FIG. 14 (e.g., a PC) is able to function as the annoyance judgmentapparatus 1 for speech sound listening according to the presentembodiment. Note that the speech sound DB 71 does not need to be storedin the memory 31, but may be stored on a hard disk (not shown) which isconnected to the bus 34.

FIG. 15 shows a functional block construction of the annoyance judgmentsystem 100 for speech sound listening of the present embodiment. Theannoyance judgment system 100 for speech sound listening includes theaudio output section 11, the biological signal measurement section 50,and the annoyance judgment apparatus 1 for speech sound listening. FIG.15 also shows detailed functional blocks of the annoyance judgmentapparatus 1 for speech sound listening. Specifically, the annoyancejudgment apparatus 1 for speech sound listening includes anevent-related potential processing section 55, an annoyance judgmentsection 65, a presented-speech sound determination section 70, a speechsound DB 71, and a result accumulating DB 80. The user 5 block isillustrated for ease of explanation.

The respective functional blocks (except the speech sound DB 71) of theannoyance judgment apparatus 1 for speech sound listening correspond tofunctions which are realized by the CPU 30, the memory 31, and the audiocontroller 32 as a whole upon executing the program which has beendescribed in conjunction with FIG. 14.

The speech sound DB 71 is a database of speech sounds which is providedfor performing annoyance judgment in speech sound listening. FIG. 16shows an exemplary speech sound DB 71 in the case where 20 speech soundsof the 67S list are to be used as test speech sounds, for example. Inthe speech sound DB 71 shown in FIG. 16, an audio file and a consonantlabel of each speech sound for presentation, as well as the referencelatency of an N1 component for each speech sound, are retained inassociation. Preferably, the reference N1 component latency of eachspeech sound is a length which is adapted to the duration or intensityof the consonant that is contained in the speech sound. The audio filesmay be standard test audios of the 20 words in the 67S list, or audiorecordings of the voice of a person who is a main partner ofconversation through a hearing aid, for example. As for the storedaudios, it is assumed that the gain adjustment (acoustic aiding process)for each frequency has been completed based on a fitting theory fromaudiograms of people suffering from hypacusia that were measured inadvance. Note that the types of speech sounds to be stored may be the 50sounds of the 57S list, instead of the 20 speech sounds of the 67S list.The consonant labels are utilized when assessing a consonant that incursa high annoyance by the user 5.

The reference N1 component latency is a threshold value (in units of ms)for the latency of an N1 component of each speech sound which accountsfor the influences of differing characteristic features of speech soundaudios. In the case of using standard test audios, the latency of an N1component that is measured for each speech sound at a sound pressurewhere a generic user feels annoyance may be used. Without being limitedto standard test audios, in the case of using audio recordings of thevoice of a person who is a main partner of conversation through thehearing aid, for example, a value which is calculated based on theconsonant duration and consonant intensity of each speech sound forpresentation may be set, for example. Through a comparison between thisreference N1 component latency and the measured N1 component latency,annoyance is judged. The method of annoyance judgment will be describedlater.

FIG. 15 is again referred to. The presented-speech sound determinationsection 70 determines which speech sound is to be presented at whatsound pressure, by referring to the speech sound DB 71. The speechsounds for presentation may be selected and determined by random order,for example. It is assumed that the sound pressures of the speech soundsfor presentation are sound pressures which are obtained after applyingan acoustic aiding process to audios of 55 dB SPL, 65 dB SPL, 75 dB SPL,and 85 dB SPL in a speech sound intelligibility curve measurement. Thesound pressures may sequentially varied from smaller sound pressures tolarger sound pressures, or in the opposite order of this. Alternatively,sound pressures may be selected by random order.

In accordance with the point of audio presentation, the presented-speechsound determination section 70 outputs a trigger to the biologicalsignal measurement section 50, and sends the actual audio to bepresented to the event-related potential processing section 55.

The audio output section 11 reproduces the monosyllabic audio which isdetermined by the presented-speech sound determination section 70,thereby presenting it to the user 5.

The biological signal measurement section 50, which is anelectroencephalograph for measuring a biological signal of the user 5,measures an electroencephalogram as the biological signal. Then, thebiological signal measurement section 50 subjects theelectroencephalogram data to a frequency filtering with a cutofffrequency which is suitable for N1 component extraction, cuts out anevent-related potential of the electroencephalogram in a predeterminedzone (e.g., a zone from −200 ms to 500 ms) based on the trigger receivedfrom the presented-speech sound determination section 70 as a startingpoint, and sends the waveform data (electroencephalogram data) thereofto the event-related potential processing section 55. The N1 componentfrequency is about 10 Hz. Therefore, in the case of using a band-passfilter as the frequency filter, it may be set so as to allow a componentfrom 5 Hz to 15 Hz of the electroencephalogram to pass through, forexample. It is assumed that the user 5 has already put on theelectroencephalograph. The electrode for electroencephalogrammeasurement is attached at the parietal Pz, for example.

In accordance with the actual audio to be presented that is receivedfrom the presented-speech sound determination section 70, theevent-related potential processing section 55 performs a summation ofthe event-related potentials received from the biological signalmeasurement section 50. The event-related potential processing section55 may only select the event-related potentials corresponding to audiopresentations of the same speech sound, thus performing a summation ofthe event-related potentials for each speech sound type, for example.Taking a summation only of the event-related potentials for the samespeech sound makes possible an annoyance judgment for each speech sound.Since the characteristic features of speech sound audios are similarbetween speech sounds sharing the same consonant, the summation may beperformed for event-related potentials for selected speech sounds thatshare the same consonant. Alternatively, in FIG. 16, those speech soundswhose differences in reference N1 component latency are within 10 ms maybe grouped up for summation. Taking a summation of speech sounds sharingthe same consonant enables an annoyance judgment in speech soundlistening with respect to each consonant type. Taking a summation ofeach group of speech sounds whose differences in reference N1 componentlatency are small enables an annoyance judgment with respect to thatgroup. From a consonant-by-consonant summation, or a group-by-groupsummation of small differences in reference N1 component latency, asummed waveform is obtained with more than a few summations being made.Moreover, as a characteristic feature of the measuredelectroencephalogram data, the event-related potential processingsection 55 may calculate an S(signal)/N(noise), where the N1 componentis the signal, for example. Note that, although an arithmetic mean ofevent-related potentials is performed in the aforementioned experiment,the mean process is not needed when paying attention to the N1 componentlatency alone.

The event-related potential processing section 55 sends theelectroencephalogram data which has been obtained by performing asummation over a predetermined number of times for each speech sound tothe annoyance judgment section 65.

Having received the electroencephalogram data from the event-relatedpotential processing section 55, the annoyance judgment section 65performs an analysis process described below.

Based on the latency of an N1 component in the electroencephalogram datareceived from the event-related potential processing section 55, theannoyance judgment section 65 judges whether the user felt annoyed ornot. For example, as the N1 component latency, the annoyance judgmentsection 65 compares a point in time at which a negative potential peakoccurs, between 50 ms and 350 ms based on a trigger received from thepresented-speech sound determination section 70 as a starting point(hereinafter referred to as a “peak latency”), against the predeterminedreference latency (threshold value) which is stored in the speech soundDB 71. Then, if the peak latency of the N1 component is shorter than thepredetermined threshold value, an “annoying” judgment is made, and ifthe peak latency is longer than the predetermined threshold value a “notannoying” judgment is made. As used herein, “annoying” or “beingannoyed” means that the speech sound has such a large sound pressurethat the user feels uncomfortable. On the other hand, “not annoying” or“not being annoyed” means that the sound pressure of the speech sound isin a range where it is not so loud as to allow the user to feeluncomfortable. Note that, without being limited to a dichotomic judgmentof either “annoying” or “not annoying”, the annoyance judgment section65 may determine a difference between the peak latency of the N1component and the reference latency.

In the case of judging annoyance for each speech sound, for example, areference latency for each speech sound may be used as the predeterminedthreshold value. In the case of judging annoyance for each row havingthe same consonant, a reference latency for each row may be used as thepredetermined threshold value. In the case of judging annoyance for eachgroup whose difference in reference latency is small, a referencelatency for each group may be used as the predetermined threshold value.

The result accumulating DB 80 receives information of the presentedaudio from the presented-speech sound determination section 70. Also,the result accumulating DB 80 receives information of the annoyancejudgment result for each speech sound from the annoyance judgmentsection 65. Then, with respect to each speech sound and each soundpressure of the presented audio, for example, the result accumulating DB80 accumulates information of the received results of annoyancejudgment.

FIG. 17 shows an example of data accumulation in the result accumulatingDB 80. FIG. 17 illustrates an example where annoyance information isaccumulated with respect to each speech sound and each sound pressure.For example, in FIG. 17, “1” denotes a case where the annoyance judgmentsection 65 determines that the N1 component latency is shorter than thereference latency, thus making an “annoying” judgment, whereas “0”denotes a case where the annoyance judgment section 65 determines thatthe N1 component latency is longer than the reference latency, thusmaking a “not annoying” judgment.

FIGS. 18A to 18C show measurement results of speech soundintelligibility curves (conventional assessment), and in addition to theconventional assessment, exemplary results of an annoyance judgment inspeech sound listening according to the present embodiment. FIG. 18Ashows an example where only speech sound intelligibility with respect toeach sound pressure of the presented audio is assessed when wearing ahearing aid. This example illustrates an assessment result which isobtained by a conventional assessment technique. In this example, theintelligibility is assessed to be 80% or more at a sound pressure of 65dB SPL or more. Therefore, if the speech sound intelligibility isimproved as compared to when not wearing a hearing aid (not shown inFIG. 18), a hearing aid suitability test would determine this case to besuitable.

FIGS. 18B and 18C show exemplary assessment results where annoyancejudgment according to the present embodiment is employed in addition tothe result of speech sound intelligibility curve measurement whenwearing a hearing aid as shown in FIG. 18A. It is assumed that thespeech sound intelligibility curve has been separately measured by aconventional method which involves oral explanation, key inputting, orthe like.

In FIGS. 18B and 18C, although the intelligibilities are the same,results of annoyance judgment are quite different. For example, in FIG.18B, the overall assessment of annoyance is low; this leads to anassessment that the user would feel little annoyance with the acousticaiding process. In FIG. 18C, for example, the overall assessment ofannoyance is high, and particularly high at large sound pressures of 65dB SPL or more; this leads to an assessment that, with this acousticaiding process, annoyance would be felt at 65 dB SPL, which is the soundpressure of daily conversations. Such assessments permits proposals ofspecific fitting procedures to be made, e.g., uniformly increasing theamount of gain adjustment in the case of FIG. 18B if the user requestsfurther improvement of intelligibility, or uniformly decreasing theamount of gain adjustment in the case of FIG. 18C and employing enhancedcompression in non-linear amplification.

Although FIGS. 18B and 18C illustrate annoyance assessment only whenwearing a hearing aid, annoyance may be assessed also when not wearing ahearing aid (naked ear), and an annoyance comparison may be made betweenwhen not wearing a hearing aid and when wearing a hearing aid.

Next, with reference to FIG. 19, a processing procedure performed by theannoyance judgment system 100 for speech sound listening of FIG. 15 willbe described. FIG. 19 is a flowchart showing a procedure of processingperformed by the annoyance judgment system 100 for speech soundlistening.

At step S101, by referring to the speech sound DB 71, thepresented-speech sound determination section 70 determines amonosyllabic speech sound to be presented and a sound pressure. Theaudio output section 11 presents the speech sound to the user 5 at thedetermined sound pressure.

The presented-speech sound determination section 70 sends a trigger tothe biological signal measurement section 50, and sends audioinformation concerning the presented speech sound to the event-relatedpotential processing section 55. The speech sound to be presented may berandomly selected from the speech sound DB 71, or a speech sound of aparticular consonant or a group may be exclusively selected. The soundpressure of the speech sound for presentation may be, for example, soundpressures which are obtained after applying an acoustic aiding processto audios of 55 dB SPL, 65 dB SPL, 75 dB SPL, and 85 dB SPL in a speechsound intelligibility curve measurement. The sound pressures ofpresentation may sequentially varied from smaller sound pressures tolarger sound pressures, or in the opposite order of this. Alternatively,sound pressures may be selected by random order.

At step S102, upon receiving the trigger from the presented-speech sounddetermination section 70, the biological signal measurement section 50cuts out an event-related potential from e.g. −200 ms to 500 ms from themeasured electroencephalogram, based on the trigger as a starting point.Then, an average potential from e.g. −200 ms to 0 ms is determined, andthe resultant event-related potential is subjected to baselinecorrection so that this average potential becomes 0 μV.

It is assumed that the biological signal measurement section 50 isalways measuring an electroencephalogram during assessment, and applyinga frequency filter which is suitable for N1 component extraction to theelectroencephalogram data. For example, a suitable frequency filter maybe a band-pass filter which allows 5 Hz to 15 Hz, around the centerfrequency of 10 Hz of the N1 component, to pass. Note that baselinecorrection is not essential in the case where a high-pass filter of 5 Hzor more is applied to the electroencephalogram data, for example,because there is hardly any influence of changes in the base line atlower frequencies.

At step S103, based on the information of the presented speech soundwhich is received from the presented-speech sound determination section70, the event-related potential processing section 55 takes a summationof the event-related potential cut out at step S102 with respect to eachspeech sound and each sound pressure. Although a summation is performedwith respect to each speech sound and each sound pressure level in thepresent embodiment, it is not necessary that the summation be performedwith respect to each speech sound. For example, it may be performed withrespect to each speech sound type, e.g., for each consonant or for eachgroup of speech sounds whose difference in reference latency is small,or with respect to each sound pressure level of presentation. In otherwords, in the case where each speech sound is classified based on atleast one of the speech sound type, the consonant type, and the groupwhose difference in reference latency is small, the event-relatedpotential processing section 55 may take a summation of event-relatedpotentials of electroencephalogram signals which are obtained whenspeech sounds belonging to the same classification are presented.

At step S104, the event-related potential processing section 55determines whether the number of summations for the event-relatedpotential with respect to the speech sound presented at step S101 hasreached a predetermined number of summations or not. If the number ofsummations is less than the predetermined number of times, the processreturns to step S101 to repeat audio presentation. If the number ofsummations is equal to or greater than the predetermined number oftimes, the process proceeds to step S105. The predetermined number be 20times, for example. Note that “20 times” is a mere example, although itis a number of summations which is frequently adopted in fields whereevent-related potentials are to be measured. For example, theevent-related potential processing section 55 may calculate anS(signal)/N(noise), where the N1 component is the signal, and a numberof summations beyond which the S/N attains a certain level or more maybe chosen as the predetermined number of times.

At step S105, the event-related potential processing section 55 sendsthe electroencephalogram data obtained by taking a summation over apredetermined number of times to the annoyance judgment section 65.

At step S106, the annoyance judgment section 65 determines the N1component latency of the electroencephalogram data received from theevent-related potential processing section 55, and compares it againstthe reference N1 component latency received from the speech sound DB 71.The N1 component latency of the electroencephalogram data may be a pointin time at which the potential becomes smallest, within a zone from 0 msto 500 ms, for example. In the case where a summation with respect toeach speech sound and each sound pressure has been performed at stepS103, a comparison between the reference latency for each speech soundand the N1 component latency of the electroencephalogram data is made.

At step S107, if the N1 component latency of the electroencephalogramdata is shorter than the reference latency, the annoyance judgmentsection 65 judges that the user 5 has felt annoyed. On the other hand,if the N1 component latency of the electroencephalogram data is longerthan the reference latency, the annoyance judgment section 65 judgesthat the user 5 has felt not annoyed.

At step S108, the result accumulating DB 80 accumulates information ofthe annoyance judgment result received from the annoyance judgmentsection 65, with respect to each speech sound and each sound pressurepresented at step S101.

At step S109, the presented-speech sound determination section 70determines whether stimulation presentation has been completed for allof the speech sounds and sound pressures to be subjected to anassessment of annoyance in speech sound listening. If it is notcompleted, the process returns to step S101; if it is completed, theannoyance judgment in speech sound listening is ended.

From the results of annoyance judgment which are accumulated in theresult accumulating DB 80 with respect to each speech sound and eachsound pressure, proposals of more specific fitting procedures can bemade. For example, if a result of speech sound intelligibility curvemeasurement as shown in FIG. 18B is obtained, where the overallassessment of annoyance is considered to be low, the amount of gainadjustment may be uniformly increased if the user requests furtherimprovement in intelligibility. As a result, a fitting which is moreappropriate for that user can be realized. On the other hand, if aresult of speech sound intelligibility curve measurement as shown inFIG. 18C is obtained, a fitting may be proposed which involves uniformlydecreasing the amount of gain adjustment and further employing enhancedcompression in non-linear amplification.

In the present embodiment, on the premise of presenting a monosyllabicspeech sound in the form of an audio, annoyance in speech soundlistening is assessed through a process which utilizes a negativecomponent at a latency of about 200 ms, of an event-related potentialbased on the point of audio presentation as a starting point. Throughthe above process, in speech sound listening, a judgment can be made asto how annoyed the user was (annoyance). This means that suitability ofan acoustic aiding process can be assessed from the perspective ofannoyance in speech sound listening, which is distinct from speech soundintelligibility. Since acoustic aiding process assessments are possiblefrom the perspective of annoyance, an acoustic aiding process which doesnot cause the user to feel annoyed in speech sound listening and whichdoes not induce aural fatigue can be realized.

Note that, as shown in FIG. 14, the annoyance judgment apparatus 1 forspeech sound listening in the present embodiment is realized with aconstruction which permits downsizing and which employs generichardware. By constructing the annoyance judgment apparatus 1 in aportable size and weight that allows the user to carry it with himselfor herself, it becomes possible to assess a comfortableness of speechsound listening in an acoustic environment in which the user actuallyuses a hearing aid. Although the audio output section 11 is illustratedas a speaker set in FIG. 13, the audio output section 11 may instead beheadphones. Use of headphones facilitates transportation, thus enablingan assessment of speech sound listening in an environment in which theuser uses them.

The present embodiment has been illustrated based on assessments for theJapanese language. However, it may be English or Chinese so long as thespeech sounds are monosyllabic. In the case of English, for example,monosyllabic words may be presented, and an evaluation may be made on aword-by-word basis. A monosyllabic English word is an audio spanning ashort period of time, and is composed of a consonant(s) and a vowel.Therefore, basically similarly to the aforementioned Japanesemonosyllabic speech sounds, a reference latency can be determined foreach word based on consonant duration and consonant intensity.

FIG. 20 shows an exemplary result of assessing annoyance for differentmonosyllabic words. In FIG. 20, “1” indicates that the user feltannoyed, whereas “0” indicates that the user did not feel annoyed.

According to the annoyance judgment system 100 for speech soundlistening of the present embodiment, as a user merely hears an audio, ajudgment can be made as to how annoyed he or she was in speech soundlistening (annoyance). As a result, an “annoyance” in speech soundlistening that is felt by the user can be quantified, whereby anassessment of an acoustic aiding process can be made from theperspective of annoyance, thus permitting a fitting not inducingannoyance and aural fatigue.

In the description of the present embodiment, the biological signalmeasurement section 50 is illustrated as cutting out an event-relatedpotential in a predetermined range based on a trigger from thepresented-speech sound determination section 70 as a starting point,subjecting it to a baseline correction, and sending potential waveformdata to the event-related potential processing section 55. However, thisprocess is an example. As another process, for example, the biologicalsignal measurement section 50 may constantly measure anelectroencephalogram, cut out an event-related potential as needed bythe event-related potential processing section 55, and subject it to abaseline correction. With such a construction, the presented-speechsound determination section 70 does not need to send a trigger to thebiological signal measurement section 50, and may only send a trigger tothe event-related potential processing section 55.

Although the present embodiment illustrates that the annoyance judgmentresults are accumulated in the result accumulating DB 80, accumulationis not necessary. For example, in the case where the result accumulatingDB 80 is provided external to the annoyance judgment apparatus 1, theannoyance judgment section 65 may simply output the result of judgment.The result of judgment can be utilized as information concerningannoyance in speech sound listening.

The annoyance judgment system illustrated in the present embodimentmakes a judgment as to how annoying a speech sound was felt to be(annoyance), based on the latency of a negative component around 200 ms(more specifically, from 50 ms to 350 ms) in the userelectroencephalogram after audio presentation. By setting a differentreference latency for each speech sound, a highly precise annoyanceassessment is enabled irrespective of characteristic features such asthe duration of the consonant portion (consonant duration) and theintensity of the consonant portion (consonant intensity) of each speechsound. Through annoyance judgment in speech sound listening, an acousticaiding process can be selected which does not allow the user to feelannoyance and which is not likely to induce fatigue even if a hearingaid is worn for a long period of time.

Embodiment 2

In the annoyance judgment system 100 for speech sound listening ofEmbodiment 1, according to one type of acoustic aiding process stored inthe speech sound DB 71, annoyance in speech sound listening is judgedfor predetermined audios that have been previously adjusted, based onthe latency of an N1 component.

However, due to increasing precision in the signal processing of therecent years, acoustic aiding processing methods for realizing functionssuch as consonant emphasis, directivity, and noise reduction are underdevelopment. This imposes a limitation on searching for and identifyingan optimum acoustic aiding process based on the annoyance judgmentresults for a single acoustic aiding process alone.

Therefore, the present embodiment will illustrate an annoyance judgmentsystem for speech sound listening including an acoustic aidingprocessing section which modifies presented speech sounds into sounds tobe output through a hearing aid, and assesses annoyance for each one ofdifferent acoustic aiding processes.

FIG. 21 shows a functional block construction of an annoyance judgmentsystem 200 for speech sound listening of the present embodiment. Theannoyance judgment system 200 for speech sound listening includes anaudio output section 11, a biological signal measurement section 50, andan annoyance judgment apparatus 2 for speech sound listening. Any blockwhich has an identical counterpart in FIG. 15 is denoted by a likereference numeral, and the description thereof is omitted. The hardwareconstruction of the annoyance judgment apparatus 2 for speech soundlistening is as shown in FIG. 14. The annoyance judgment apparatus 2 forspeech sound listening of the present embodiment shown in FIG. 21 isrealized as a program which defines a different process from that of theprogram 35 described in Embodiment 1 (FIG. 14) is executed. Theannoyance judgment system 200 may also be referred to as an acousticaiding processing system.

One large difference of the annoyance judgment apparatus 2 for speechsound listening of the present embodiment from the annoyance judgmentapparatus 1 for speech sound listening of Embodiment 1 is that anacoustic aiding processing section 90 is additionally introduced.Although each component element of the annoyance judgment apparatus 2 isbasically given the same name as that used in Embodiment 1, they may bedenoted by different reference numerals when having different operationsand/or functions. For example, the present embodiment performs anannoyance judgment for each of a plurality of acoustic aiding processes,which is not performed in Embodiment 1; therefore, in the place of theevent-related potential processing section 55, the presented-speechsound determination section 70, the speech sound DB 71, and the resultaccumulating DB 80 of Embodiment 1, the present embodiment employs anevent-related potential processing section 56, a presented-speech sounddetermination section 75, a speech sound DB 72, and a resultaccumulating DB 85.

Hereinafter, the speech sound DB 72, the presented-speech sounddetermination section 75, the acoustic aiding processing section 90, theevent-related potential processing section 56, the result accumulatingDB 85, and the acoustic aiding processing section 90 will be described.

Similarly to the speech sound DB 71 of Embodiment 1, the speech sound DB72 is a speech sound database with which to conduct an annoyancejudgment in speech sound listening, as in the 20 speech sounds of the67S list shown in FIG. 16, for example. Similarly to the speech sound DB71, the speech sound DB 72 also retains information of a reference N1component latency for each speech sound. The speech sound DB 72 differsfrom the speech sound DB 71 in that the speech sound DB 72 containsspeech sound data before being subjected to an acoustic aiding process.

Similarly to the presented-speech sound determination section 70 ofEmbodiment 1, the presented-speech sound determination section 75determines a speech sound type and a sound pressure by referring to thespeech sound DB. The presented-speech sound determination sectiondiffers from the presented-speech sound determination section 70 in thatthe presented-speech sound determination section 75 allows a selectionas to which acoustic aiding process an audio is to be modified through,and it sends also the audio data of the speech sound for presentation tothe acoustic aiding processing section 90.

Based on the instruction concerning the acoustic aiding process to beselected and the audio data received from the presented-speech sounddetermination section 75, the acoustic aiding processing section 90modifies the audio data with the designated acoustic aiding processingmethod. The acoustic aiding process may involve consonant emphasis,directivity, noise reduction, etc., for example. In the case where anacoustic aiding process involving consonant emphasis is selected, forexample, a process of increasing the amount of gain amplification forconsonant frequencies than usual is performed, thus modifying the audiodata. Note that the acoustic aiding processing section 90 may rely onthe judgment result by the annoyance judgment section 65 to adjust theamount of gain amplification for audios. For example, for the audio dataof a speech sound that is judged as annoying by the annoyance judgmentsection 65, the amount of gain amplification is reduced. For the audiodata of a speech sound judged as not annoying by the annoyance judgmentsection 65, the amount of gain amplification is not adjusted.Alternatively, the amount of gain amplification may be determined on thebasis of a difference between the peak latency of the N1 component andthe reference latency as determined by the annoyance judgment section65. For example, if the difference between the peak latency of the N1component and the reference latency is within a predetermined range, theacoustic aiding processing section 90 does not adjust the amount of gainamplification. As the difference between the peak latency of the N1component and the reference latency deviates away from the upper limitvalue or lower limit value of the predetermined range, the acousticaiding processing section 90 decreases the amount of gain amplification.

Similarly to the event-related potential processing section 55 ofEmbodiment 1, in accordance with the actual audio to be presented whichis received from the presented-speech sound determination section 75,the event-related potential processing section 56 performs a summationfor the event-related potentials received from the biological signalmeasurement section 50. A difference between the event-related potentialprocessing section 56 and the event-related potential processing section55 is that, upon receiving the information of an acoustic aiding processfrom the presented-speech sound determination section 75, theevent-related potential processing section 56 performs a summation withrespect to each speech sound, each sound pressure, and each acousticaiding process.

Similarly to the result accumulating DB 80 of Embodiment 1, the resultaccumulating DB 85 accumulates information of annoyance judgment results(based on N1 component latency) received from the annoyance judgmentsection 65, e.g., with respect to each speech sound and each soundpressure. A difference between the result accumulating DB 85 and theresult accumulating DB 80 is that, from the presented-speech sounddetermination section 75, the result accumulating DB 85 receives notonly information of the speech sound and sound pressure of the presentedstimulation, but also information of an acoustic aiding process type,and accumulates the data with respect to each acoustic aiding processtype.

FIGS. 22A and 22B show examples of data accumulation in the resultaccumulating DB 85. FIGS. 22A and 22B illustrate examples where theresults of annoyance judgment are accumulated with respect to eachspeech sound, each sound pressure, and each acoustic aiding process.FIG. 22A shows a pattern under acoustic aiding process A, whereas FIG.22B shows a pattern under acoustic aiding process B. Each indicatesannoyance assessment results in the case where speech sounds aresubjected to the respective acoustic aiding process. In FIGS. 22A and22B, “1” indicates the case where the annoyance judgment section 65judges that the N1 component latency is shorter than the referencelatency and that the user 5 has felt annoyed, and “0” indicates the casewhere the annoyance judgment section 65 judges that the N1 componentlatency is longer than the reference latency and that the user 5 has notfelt annoyed. From a comparison between FIGS. 22A and 22B, it can besaid that there are fewer “1's” (i.e., the user is feeling lessannoyance) in FIG. 22B where an acoustic aiding process is performedbased on the pattern of acoustic aiding process B.

Next, with reference to FIG. 23, an overall procedure of processing thatis performed in the annoyance judgment system 200 for speech soundlistening will be described.

FIG. 23 is a flowchart showing the processing procedure by the annoyancejudgment system 200 for speech sound listening of the presentembodiment. In FIG. 23, any step where a process identical to a processby the annoyance judgment system 100 for speech sound listening (FIG.19) will be denoted by a like reference numeral, and the descriptionthereof will be omitted.

The processes by the annoyance judgment system 200 for speech soundlistening of the present embodiment differ from the processes by theannoyance judgment system 100 for speech sound listening of Embodiment 1in steps S201, S202, and S203. At step S201, a monosyllabic audio whichis modified through the designated acoustic aiding process is presented.At step S202, a summation is performed with respect to each speechsound, each sound pressure, and each acoustic aiding process. At stepS203, results are accumulated with respect to each speech sound, eachaudio, and each acoustic aiding process. Any other steps have alreadybeen described in connection with FIG. 19, and the descriptions thereofare omitted.

At step S201, the presented-speech sound determination section 75determines the type and sound pressure of the audio to be presented byreferring to the speech sound DB 72, and acquires the data thereof.Furthermore, the presented-speech sound determination section determinesan acoustic aiding process, and sends the information concerning theacoustic aiding process type and the audio data to the acoustic aidingprocessing section 90. The acoustic aiding processing section 90determines the information concerning the acoustic aiding process typeas determined by the presented-speech sound determination section 75 andthe audio data, and modifies the audio data based on the designatedacoustic aiding processing method. The audio output section 11 presentsthe modified audio data to the user 5.

At step S202, information of the type of speech sound for presentation,sound pressure, and acoustic aiding process received from thepresented-speech sound determination section 75, the event-relatedpotential processing section 56 takes a summation of the event-relatedpotential of the electroencephalogram measured by the biological signalmeasurement section 50, e.g., with respect to each speech sound, eachsound pressure, and each acoustic aiding process.

At step S203, with respect to each of the pieces of informationconcerning the speech sound for presentation (speech sound type, soundpressure, acoustic aiding process) received from the presented-speechsound determination section 75, the result accumulating DB accumulatesthe result of annoyance judgment based on the N1 component latency asdetermined by the annoyance judgment section 65. Examples of accumulatedresults are as shown in FIG. 22.

Through such processes, comfortableness in speech sound listening can beassessed, e.g., with respect to each acoustic aiding process such asconsonant emphasis, directivity, or noise reduction.

Although the present embodiment contemplates a case where audios havingbeen subjected to a plurality of types of acoustic aiding processes aremixed up and presented in random order, annoyance judgment may beperformed in sequential order among different types of acoustic aidingprocesses, e.g., acoustic aiding process A first and then acousticaiding process B, for example. Performing annoyance judgment for eachdifferent type of acoustic aiding process provides an advantage in thatthe parameters in the subsequently following acoustic aiding process canbe changed in accordance with the result of annoyance judgment.

With the annoyance judgment system 200 for speech sound listening of thepresent embodiment, annoyance for each acoustic aiding process can beassessed. As a result, selection of an acoustic aiding process which isadapted to the purpose of wearing a hearing aid and to the environmentof use can be realized.

Embodiment 3

In the annoyance judgment system 100 for speech sound listening ofEmbodiment 1, the annoyance judgment section 65 makes an annoyancejudgment through comparison between a reference N1 component latency ofa generic user for each speech sound and the latency of an N1 componentof the measured electroencephalogram data.

The N1 component is an incipient component of an event-related potentialcalled the evoked potential, and is believed to have relatively smallindividual differences concerning latency/amplitude. However, it is notthat the N1 component is completely free of latency/amplitude individualdifferences. Therefore, there has been a limit to making a highprecision annoyance judgment through identification based on a referencelatency which is obtained from the latency of an N1 component of ageneric user for each speech sound.

Accordingly, in the present embodiment, prior to annoyance judgment inspeech sound listening, a calibration is made for measuring thereference N1 component latency of each user, and annoyance is assessedbased on the N1 component traits of each individual. As a result, in thepresent embodiment, annoyance judgment can be made with a higherprecision than in Embodiment 1.

FIG. 24 shows a functional block construction of an annoyance judgmentsystem 300 for speech sound listening of the present embodiment. Theannoyance judgment system 300 for speech sound listening includes anaudio output section 11, a biological signal measurement section 50, andan annoyance judgment apparatus 3 for speech sound listening. Any blockwhich has an identical counterpart in FIG. 15 is denoted by a likereference numeral, and the description thereof is omitted. The hardwareconstruction of the annoyance judgment apparatus 3 for speech soundlistening is as shown in FIG. 14. The annoyance judgment apparatus 3 forspeech sound listening of the present embodiment shown in FIG. 24 isrealized as a program which defines a different process from that of theprogram 35 described in Embodiment 1 (FIG. 14) is executed.

One large difference of the annoyance judgment apparatus 3 for speechsound listening of the present embodiment from the annoyance judgmentapparatus 1 for speech sound listening of Embodiment 1 is that acalibration/assessment switching section 95 and a reference latencycalculation section 96 are additionally introduced. Moreover, areference N1 component latency of each user is obtained for each speechsound, and in order to perform annoyance judgment based on thisreference latency, the present embodiment employs a presented-speechsound determination section 77, an event-related potential processingsection 57, and a speech sound DB 73, in the place of thepresented-speech sound determination section 70, the event-relatedpotential processing section 55, and the speech sound DB 71 (FIG. 15) ofEmbodiment 1.

Hereinafter, the calibration/assessment switching section 95, thereference latency calculation section 96, the presented-speech sounddetermination section 77, the event-related potential processing section57, and the speech sound DB 73 will be described.

The calibration/assessment switching section 95 switches between acalibration mode for identifying a reference N1 component latency ofeach user for each speech sound, and an assessment mode of making anannoyance judgment based on the identified reference latency and ameasured N1 component latency. Then, calibration/assessment switchingsection 95 sends information representing the current mode to thepresented-speech sound determination section 77. Note that the modeswitching may be conducted at a point in time when a reference latencyfor each speech sound is written to the speech sound DB 73, or when apredetermined number of times of speech sound presentation that isrequired for identifying the reference N1 component latency of the userelectroencephalogram for each speech sound are finished.

The presented-speech sound determination section 77 refers to the speechsound DB 73 to determine a speech sound type and the sound pressure ofthe presented audio, and outputs the speech sound to the user 5 via theaudio output section 11, and also sends trigger information to thebiological signal measurement section 50. Moreover, the presented-speechsound determination section 77 receives calibration mode/assessment modeinformation from the calibration/assessment switching section 95, andsends audio information and the calibration/assessment mode informationto the event-related potential processing section 57. Thepresented-speech sound determination section 77 switches its operationin accordance with the mode received from the calibration/assessmentswitching section 95. In the calibration mode, an audio of a vowel (amonosyllabic sound with no consonant portion) may be presented as apredetermined sound pressure, for example. By presenting a vowel as thespeech sound audio, the N1 component latency to serve as a basis foreach user can be identified without the influence of a consonantportion. The predetermined sound pressure is a sound pressure which isabove a threshold value at which the user can hear an audio. Forexample, it may be a sound pressure which is felt as “annoying” by theuser. The sound pressure above which the user is able to hear and thesound pressure at which the user feels annoyed may be determined basedon the user's audiogram, or previously measured through subjectiveevaluation, for example. In the assessment mode, the presented-speechsound determination section 77 presents a speech sound at apredetermined sound pressure, similarly to the presented-speech sounddetermination section 70.

Similarly to the event-related potential processing section 55, theevent-related potential processing section 57 takes a summation of theevent-related potentials received from the biological signal measurementsection 50, in accordance with the actual audio to be presented which isreceived from the presented-speech sound determination section 77.Moreover, the event-related potential processing section 57 switches itsoperation in accordance with mode information received from thepresented-speech sound determination section 77. In the calibrationmode, a summation is taken with respect to each vowel, for example, andafter a predetermined number of times of summation are finished, thesummed waveform with respect to each vowel is sent to the referencelatency calculation section 96. In the assessment mode, similarly to theevent-related potential processing section 55, a summed waveform withrespect to each speech sound and each sound pressure is sent to theannoyance judgment section 65.

Receiving the summed waveform data with respect to each vowel from theevent-related potential processing section 57, the reference latencycalculation section 96 determines the latency of an N1 component. The N1component latency may be a point in time at which the potential becomessmallest within a range from 50 ms to 350 ms, for example. For example,if the presented-speech sound determination section 77 has set a soundpressure which is felt as “annoying” by the user as the sound pressure,the N1 component latency determined with respect to each vowel isregarded as the reference latency for the respective vowel. By measuringan N1 component latency for the vowel, it becomes possible to determinea reference N1 component latency for each vowel in accordance with thediffering sound characteristic features of different vowels. Thereference latency for any speech sound containing a consonant portion isobtained by adding a predetermined positive value, which is adapted tothe characteristic feature of the consonant portion, to the referencelatency for each vowel. The predetermined positive value is determinedfor each consonant. For example, in the case of a speech sound whoseconsonant portion has a weak intensity, the consonant duration may beused as the predetermined positive value. In the case of a speech soundwhose consonant portion has a strong intensity, for example, the timewhich elapses until the intensity of the consonant portion becomes equalto or greater than a predetermined value may be used as thepredetermined positive value. Then, the calculated result is written tothe speech sound DB 73.

As compared to consonants, differences in audio characteristic featuresfrom speech sound to speech sound are smaller among vowels. Therefore, amean of N1 component latencies for all vowels may be taken, which may beused as a reference N1 component latency for vowels. Alternatively, onlya mean of the latencies for those vowels which permitted stablemeasurement of an N1 component may be taken, and this may be used as areference N1 component latency for vowels.

The speech sound DB 73 is a database of speech sounds for use in theannoyance judgment in speech sound listening, similarly to the speechsound DB 71 exemplified in FIG. 16. A difference between the speechsound DB 73 and the speech sound DB 71 is that the speech sound DB 73permits rewrite of reference N1 component latencies. Until the referencelatency calculation section 96 rewrites the reference N1 componentlatency, indicating that no reference latency has been set may beretained for each speech sound.

Next, with reference to the flowchart of FIG. 25, an overall procedureof processing performed by the annoyance judgment system 300 for speechsound listening will be described.

FIG. 25 is a flowchart showing the processing procedure by the annoyancejudgment system 300 of the present embodiment. In FIG. 25, any stepwhere a process identical to a process by the annoyance judgment system100 for speech sound listening (FIG. 19) will be denoted by a likereference numeral, and the description thereof will be omitted.

The processes by the annoyance judgment system 300 for speech soundlistening of the present embodiment differ from the processes by theannoyance judgment system 100 for speech sound listening of Embodiment 1in steps S301 to S306. Any other steps have already been described inconnection with FIG. 19, and the descriptions thereof are omitted.

At step S301, the calibration/assessment switching section 95 sets thecurrent mode to the calibration mode, and sends information indicatingthe calibration mode to the presented-speech sound determination section77. The calibration/assessment switching section 95 may refer to thespeech sound DB 73, and if the value of the reference N1 componentlatency is 0, select the calibration mode. Moreover, the calibrationmode may be allowed to stand until a predetermined number of times audiopresentation are finished. Note that the switching between thecalibration and assessment modes may be performed by a hearing aidfitting expert or be based on selections by the user 5.

At step S302, the presented-speech sound determination section 77 refersto the speech sound DB 73 to select a vowel, for example, and outputs itat a predetermined sound pressure to the user 5 via the audio outputsection 11. The predetermined sound pressure is a sound pressure whichis above a threshold value at which the user can hear an audio. Forexample, it may be a sound pressure which is felt as “annoying” by theuser. The sound pressure above which the user is able to hear and thesound pressure at which the user feels annoyed may be determined basedon the user's audiogram.

At step S303, the event-related potential processing section 57 takes asummation of the event-related potential measured by the biologicalsignal measurement section 50 with respect to each vowel.

At step S304, from the event-related potential processing section 57,the reference latency calculation section 96 receives waveform dataafter vowel-for-vowel summation, and determines the latency of an N1component. The N1 component latency may be a point in time at which thepotential becomes smallest within a range from 50 ms to 350 ms, forexample. For example, if the presented-speech sound determinationsection 77 has set a sound pressure which is felt as “annoying” by theuser as the sound pressure, the N1 component latency determined withrespect to each vowel is regarded as the reference latency for therespective vowel. The reference latency for any speech sound containinga consonant portion is obtained by adding a predetermined positivevalue, which is adapted to the characteristic feature of the consonantportion, to the reference latency for each vowel. The predeterminedpositive value is determined for each consonant. For example, in thecase of a speech sound whose consonant portion has a weak intensity, theconsonant duration may be used as the predetermined positive value. Inthe case of a speech sound whose consonant portion has a strongintensity, for example, the time which elapses until the intensity ofthe consonant portion becomes equal to or greater than a predeterminedvalue may be used as the predetermined positive value.

At step S305, the reference latency calculation section 96 writes thereference N1 component latency for each speech sound calculated at stepS304 to the speech sound DB 73.

At step S306, upon detecting that a reference N1 component latency hasbeen written to the speech sound DB 73, the calibration/assessmentswitching section 95 switches from the calibration mode to theassessment mode, and sends information indicating the assessment mode tothe presented-speech sound determination section 77. The mode switchingmay occur when a predetermined number of times of audio presentation arefinished, or be made based on a control input by a hearing aid fittingexpert or the user 5.

Through such processes, it becomes possible compare the reference N1component latency of each user for each speech sound and the latency ofan N1 component of the measured electroencephalogram data, wherebyannoyance in speech sound listening can be assessed with a higherprecision.

In the present embodiment, a vowel(s) is selected by thepresented-speech sound determination section 77 in the calibration mode,and a reference latency for the vowel(s) is determined based on an N1component latency(s) for the vowel(s), from which reference latenciesfor consonants are calculated. By doing so, it becomes unnecessary tocalculate reference latencies for all speech sounds, whereby a test canbe conducted in a short period of time. However, this is only anexample. For example, all speech sounds to be subjected to annoyancejudgment may be presented in the calibration mode, and referencelatencies may be determined for all of the speech sounds. Alternatively,for example, pure tones which are generally measured in audiograms,e.g., 250 Hz, 500 Hz, 1 kHz, 2 kHz, and 4 kHz may be retained in thepresented-speech sound determination section 77, and these pure tonesmay be presented in the calibration mode, and a reference latency foreach speech sound may be calculated from the N1 component latencies forthe pure tones.

With the annoyance judgment system 300 for speech sound listening of thepresent embodiment, annoyance can be highly accurately assessed inaccordance with the electroencephalographic traits of each user. As aresult, an acoustic aiding process which does not induce annoyance andaural fatigue in the user can be realized.

Although Embodiments 1 to 3 above illustrate that the speech sound DB isprovided in the annoyance judgment apparatus for speech sound listening,this is not a requirement. The speech sound. DB may be provided in adatabase server (not shown) or an external storage device which isconnected to the annoyance judgment apparatus for speech sound listeningvia a network, for example. In that case, the annoyance judgment systemfor speech sound listening of each embodiment includes the databaseserver or external storage device.

Embodiment 4

In the annoyance judgment system 100 for speech sound listening ofEmbodiment 1, speech sound DB 71 recorded audios and reference N1component latencies for speech sounds, each of which is adapted to thecharacteristic feature of the respective speech sound audio, areretained, and an annoyance judgment is made through a comparison of thelatency of an N1 component of the measured electroencephalogram data.The recorded audios may be standard test audios or audios from a personwho is a main partner of conversation through the hearing aid(hereinafter referred to as “speaker A”), for example. Annoyancedetermination for audios which are utterances of speaker A pertains toan assessment of audios with which speaker A speaks to the user in dailylife, and is important.

However, recording audios of speaker A prior to annoyance judgment istroublesome to both speaker A and a hearing aid fitting expert.

Therefore, in the present embodiment, speech sound audios uttered byspeaker A are analyzed in real time; a reference N1 component latency ofthe user 5 is estimated in accordance with the characteristic feature ofthe speech sound audios of speaker A, and annoyance is assessed throughcomparison between the estimated reference latency and an N1 componentlatency of the measured electroencephalogram.

FIG. 26 shows the functional block construction of an annoyance judgmentsystem 400 for speech sound listening of the present embodiment. Theannoyance judgment system 400 for speech sound listening includes acharacter output section 12, an audio input section 41, a biologicalsignal measurement section 50, and an annoyance judgment apparatus 4 forspeech sound listening. Any block which has an identical counterpart inFIG. 15 is denoted by a like reference numeral, and the descriptionthereof is omitted.

The annoyance judgment system 400 for speech sound listening of thepresent embodiment differs from the annoyance judgment system 100 forspeech sound listening of Embodiment 1 in that the audio input section41 is additionally introduced, and that the character output section 12is provided instead of the audio output section 11. Due to the additionof these component elements, the annoyance judgment apparatus 4 forspeech sound listening has different functionality than that of theannoyance judgment apparatus 1 for speech sound listening.

The character output section 12 is a display device which outputs textinformation of a speech sound to speaker A, e.g., a liquid crystaldisplay. As the text information, monosyllabic speech sounds to beuttered by speaker A (e.g., “

(a)”, “

(da)”, “

(shi)”) are presented. In addition to the monosyllabic speech sounds,information of a sound pressure concerning how loud speaker A issupposed to make utterances may be indicated. Information concerningsound pressure may be “in a usual speaking voice”, “in a loud voice”, or“in a soft voice”, for example.

The audio input section 41 is a microphone with which to collect theaudios as uttered by speaker A.

The annoyance judgment apparatus 4 for speech sound listening will bedescribed later.

FIG. 27 shows the hardware construction of the annoyance judgmentapparatus 4 for speech sound listening. Any constituent element whichhas an identical counterpart in Embodiment 1 as shown in FIG. 14 isdenoted by a like reference numeral, and the description thereof isomitted. Differences from FIG. 14 are that: a sound controller 46 isadditionally introduced; a graphics controller 45 is provided instead ofthe audio controller 32; and the speech sound DB 71 is omitted from thememory 31.

The sound controller 46 subjects an audio waveform which has beenuttered by speaker A and input via the audio input section 41 to an A/Dconversion, and sends the resultant digital audio data to the CPU 30.

In accordance with an instruction from the CPU 30, the graphicscontroller 45 outputs text information of the speech sound forpresentation to the character output section 12.

The processes by the annoyance judgment apparatus 4 for speech soundlistening of the present embodiment, e.g., analysis of the audiowaveform, are realized as a program which defines a different processfrom that of the program 35 described in Embodiment 1 (FIG. 14) isexecuted.

FIG. 26 is referred to again. One large difference of the annoyancejudgment apparatus 4 for speech sound listening of the presentembodiment from the annoyance judgment apparatus 1 for speech soundlistening of Embodiment 1 is that an audio analysis section 42 and areference latency estimation section 43 are additionally introduced.Moreover, in the present embodiment, the presented-speech sounddetermination section 78 and the annoyance judgment section 66 determinea speech sound type to be uttered by speaker A, determines a referenceN1 component latency for the speech sound audio uttered by speaker A,and performs an annoyance judgment based on that reference latency.

Hereinafter, the presented-speech sound determination section 78, theaudio analysis section 42, the reference latency estimation section 43,and the annoyance judgment section 66 will be described.

The presented-speech sound determination section 78 refers to apreviously provided speech sound list which is retained therein torandomly determine a speech sound to be uttered by speaker A, andoutputs it to speaker A via the character output section 12. Then, thepresented-speech sound determination section 78 sends the information ofthe determined speech sound to the event-related potential processingsection 55 and the reference latency estimation section 43. The speechsound list may be the 20 speech sounds of the 67S list, or the 50 soundsof the 57S list, for example.

From the sound information having been input to the audio input section41, the audio analysis section 42 detects the timing with which speakerA uttered the audio, and analyzes characteristic features concerning theaudio, e.g., the consonant duration, the consonant intensity, the vowelintensity. Then, at the timing of detecting the utterance of speaker A,the audio analysis section 42 sends a trigger to the biological signalmeasurement section 50. Furthermore, the audio analysis section 42 sendsinformation concerning the audio characteristic features to thereference latency estimation section 42.

Based on the information concerning the audio characteristic featuresreceived from the audio analysis section 42 and the speech soundinformation received from the presented-speech sound determinationsection 78, the reference latency estimation section 43 estimates areference N1 component latency for that speech sound.

Similarly to the annoyance judgment section 65 of Embodiment 1, based onthe latency of an N1 component in the electroencephalogram data receivedfrom the event-related potential processing section 55, the annoyancejudgment section 66 judges whether the user felt annoyed or not. Forexample, the annoyance judgment section 66 compares the peak latency ofa negative potential at a latency from 50 ms to 350 ms against thereference latency (threshold value) which has been estimated by thereference latency estimation section 43. Then, if the peak latency ofthe N1 component is shorter than the threshold value, an “annoying”judgment is made; and if the peak latency is longer than thepredetermined threshold value, a “not annoying” judgment is made.

Next, with reference to the flowchart of FIG. 28, an overall procedureof processing performed by the annoyance judgment system 400 for speechsound listening will be described.

FIG. 28 shows a processing procedure by the speech sound intelligibilitysystem 400 of the present embodiment. In FIG. 28, any step where aprocess identical to a process (FIG. 19) by the annoyance judgmentsystem 100 for speech sound listening will be denoted by a referencenumeral, and the description thereof will be omitted.

The processes by the annoyance judgment system 400 for speech soundlistening of the present embodiment differ from the processes by theannoyance judgment system 100 for speech sound listening of Embodiment 1in steps S401 to S407. Any other steps have already been described inconnection with FIG. 19, and the descriptions thereof are omitted.

At step S401, referring to the speech sound list retained in thepresented-speech sound determination section 78, the presented-speechsound determination section 78 randomly determines a speech sound typeto be uttered by speaker A, and outputs the determined speech sound tospeaker A via the character output section 12.

At step S402, from the sound information having been input to the audioinput section 41, the audio analysis section 42 detects the timing withwhich speaker A uttered the audio, and at the timing of detection, sendsa trigger to the biological signal measurement section 50.

At step S403, the audio analysis section 42 analyzes the characteristicfeatures of the audio detected from the sound information input to theaudio input section 41, e.g., the consonant duration, the consonantintensity, and the vowel intensity. Then, the audio analysis section 42sends the analysis results to the reference latency estimation section43.

At step S404, based on the information concerning the audiocharacteristic features received from the audio analysis section 42 andthe speech sound information received from the presented-speech sounddetermination section 78, the reference latency estimation section 43estimates a reference N1 component latency for that speech sound. Thereference latency is estimated by adding a predetermined positive value,which is based on the consonant duration or consonant intensity of theaudio, to a predetermined base latency. The predetermined base latencymay be the latency of an average N1 component of a generic user whenhearing a vowel at 90 dB SPL, for example. Specifically, it may be 100ms. Moreover, the predetermined positive value is determined for eachconsonant. For example, in the case of a speech sound whose consonantportion has a weak intensity, the consonant duration having beenanalyzed by the audio analysis section 42 may be regarded as thepredetermined positive value. In the case of a speech sound whoseconsonant portion has a strong intensity, the intensity of the consonantportion having been analyzed by the audio analysis section 42 or thetime which elapses until the intensity of a specific frequency in theconsonant portion becomes equal to or greater than a predetermined valuemay be used as the predetermined positive value. The relative strengthof the consonant intensity may be determined based on the speech soundinformation received from the presented-speech sound determinationsection 78. Then, the estimated reference latency is sent to theannoyance judgment section 66.

At step S405, the event-related potential processing section 55 sendsthe actual audio to be presented that has been received from thepresented-speech sound determination section 70 and the event-relatedpotential that has been received from the biological signal measurementsection 50 to the annoyance judgment section 66.

At step S406, based on the latency of an N1 component in theelectroencephalogram data received from the event-related potentialprocessing section 55, the annoyance judgment section 66 judges whetherthe user felt annoyed or not. For example, the annoyance judgmentsection 65 compares the peak latency of a negative potential at alatency from 50 ms to 350 ms against the reference latency received fromthe reference latency estimation section 43. Then, if the peak latencyof the N1 component is shorter than the reference latency, an “annoying”judgment is made; and if the peak latency is longer than the referencelatency, a “not annoying” judgment is made.

At step S407, the result accumulating DB 80 receives the speech soundtype from the presented-speech sound determination section 77, andreceives information of the annoyance judgment result from the annoyancejudgment section 66. Then, accumulates the information of the annoyancejudgment result for each speech sound, for example.

Through such processes, the characteristic features of an audio whichhas been uttered by speaker A are analyzed; a reference N1 componentlatency is estimated for each audio characteristic feature; and acomparison between the latency of an N1 component of the measuredelectroencephalogram data and the estimated reference N1 componentlatency can be made. In the case where speaker A is allowed to freelyutter monosyllabic speech sounds, while the audio analysis section 42recognizes the audios, the presented-speech sound determination section78 and the character output section 12 may be omitted.

In accordance with the annoyance judgment system 400 for speech soundlistening of the present embodiment, annoyance judgment in speech soundlistening can be realized in real time by using audios uttered byspeaker A. As a result, an acoustic aiding process which does not induceannoyance and aural fatigue in the user can be realized, without muchtrouble being incurred by speaker A and a hearing aid fitting expert.

Moreover, in the present embodiment, instead of sending a trigger to thebiological signal measurement section 50, the audio analysis section 42may send a trigger to the event-related potential processing section 55.In this case, the biological signal measurement section 50 mayconstantly measure an electroencephalogram, cut out an event-relatedpotential as needed by the event-related potential processing section55, and subject it to a baseline correction.

In the above-described embodiments, the annoyance judgment section isillustrated as performing an operation of “judging”, and thepresented-speech sound determination section as performing an operationof “determining”, for example. However, these expressions are employedfor facilitating human understanding of the respective operations, andit is not intended that the apparatus actually has the discretion in“judging” and “determining” what is to be externally output. The“annoyance judgment section” or the “presented-speech sounddetermination section”, as a component element of the apparatus, maysimply perform a certain process when certain conditions are met.

For example, when the N1 component latency of the electroencephalogramdata is shorter than a reference latency, the annoyance judgment sectionmay accumulate this result under the classification that the user feltannoyed, or when the N1 component latency of the electroencephalogramdata is longer than the reference latency, the annoyance judgmentsection may accumulate this result under the classification that theuser has not felt annoyed. On the other hand, the presented-speech sounddetermination section may cause speech sounds to be selected in randomorder by referring to the speech sound DB, and a sound pressure to berandomly selected for output at the audio output section. Such processesare conveniently described by using the expressions “judge” and“determine”.

The above explanation also applies when the process according to thepresent disclosure is executed as a method.

With the annoyance judgment apparatus for speech sound listeningaccording to the present disclosure and an annoyance judgment system forspeech sound listening which incorporates the annoyance judgmentapparatus for speech sound listening, in addition to the intelligibilityas to whether a speech sound has been aurally distinguished or not, aquantitative annoyance judgment in speech sound listening can be madebased on an electroencephalogram when listening to the speech sound. Asa result, an acoustic aiding process which does not induce annoyance andaural fatigue in the user can be selected. Thus, the technique disclosedherein is applicable to the fitting for any and all hearing aid users.

While the present invention has been described with respect to preferredembodiments thereof, it will be apparent to those skilled in the artthat the disclosed invention may be modified in numerous ways and mayassume many embodiments other than those specifically described above.Accordingly, it is intended by the appended claims to cover allmodifications of the invention that fall within the true spirit andscope of the invention.

1. An annoyance judgment system comprising: a biological signalmeasurement section for measuring an electroencephalogram signal of auser; a speech sound database retaining a plurality of monosyllabicspeech sounds such that, for each speech sound, the speech sound and areference latency of an electroencephalogram negative componentcorresponding to the speech sound are retained in association; apresented-speech sound determination section configured to determine amonosyllabic speech sound to be presented by referring to the speechsound database; an output section configured to present the determinedspeech sound to the user; and an annoyance judgment section configuredto judge annoyance of the output speech sound by comparing a peaklatency of a negative component of the electroencephalogram signal in arange from 50 ms to 350 ms from a starting point, the starting pointbeing a point in time at which the speech sound is presented, againstthe reference latency corresponding to the determined speech sound thatis retained in the speech sound database.
 2. The annoyance judgmentsystem of claim 1, wherein the speech sound database keeps theassociation between each speech sound and a reference latency of anelectroencephalogram negative component corresponding to the speechsound on the basis of a duration or intensity of a consonant containedin the speech sound.
 3. The annoyance judgment system of claim 1,wherein, if the peak latency of the negative component is equal to orsmaller than the reference latency, the annoyance judgment sectionjudges that the audio signal is annoying to the user, and if the peaklatency of the negative component is greater than the reference latency,the annoyance judgment section judges that the audio signal is notannoying to the user.
 4. The annoyance judgment system of claim 1,further comprising an event-related potential processing sectionconfigured to take a summation of event-related potentials of theelectroencephalogram signal according to a predetermined criterion andconfigured to output a result of summation to the annoyance judgmentsection, wherein, the presented-speech sound determination sectiondetermines two or more speech sounds; the output section sequentiallypresents the determined speech sounds; and among the determined speechsounds, the event-related potential processing section takes a summationof event-related potentials for speech sounds of a same speech soundtype or a same sound pressure, each event-related potential being basedon a point in time of presenting the respective speech sound as astarting point.
 5. The annoyance judgment system of claim 1, wherein, asthe peak latency, the annoyance judgment section adopts: a point in timeat which a negative component of the electroencephalogram signal in arange from 50 ms to 350 ms from a starting point takes a smallestpotential, the starting point being a point in time at which thedetermined speech sound is presented; or a peak latency that isassociated with a template having a highest degree of matching, amongpreviously-provided templates of N1 component waveforms, with theelectroencephalogram signal.
 6. The annoyance judgment system of claim5, wherein the event-related potential processing section takes asummation of event-related potentials with respect to each consonant, orwith respect to each group of speech sounds whose differences inreference latency is smaller than a predetermined value.
 7. Theannoyance judgment system of claim 1, further comprising a resultaccumulating database configured to accumulate information indicating aresult of annoyance judgment for the speech sound, wherein the resultaccumulating database accumulates information indicating the result ofannoyance judgment for the speech sound with respect to each speechsound, each consonant, or each group of speech sounds whose differencesin reference latency is smaller than a predetermined value.
 8. Theannoyance judgment system of claim 1, further comprising an acousticaiding processing section configured to select a type of acoustic aidingprocess for the presented speech sound, and modifies data of the speechsounds retained in the speech sound database based on the selectedacoustic aiding process.
 9. The annoyance judgment system of claim 1,further comprising a switching section configured to switch between acalibration mode of determining reference latencies of negativecomponents for the user and an assessment mode of assessing annoyance,wherein, in the calibration mode, the switching section causes thepresented-speech sound determination section to select a vowel, andcalculates a reference latency for each speech sound based on a latencyof the negative component for the vowel; and after switching to theassessment mode, the switching section causes the annoyance judgmentsection to compare the peak latency of the negative component againstthe calculated reference latency.
 10. The annoyance judgment system ofclaim 9, wherein, in the calibration mode, when a vowel is selected bythe presented-speech sound determination section, the switching sectionsets a latency of an N1 component for the vowel as a reference latencyfor the vowel, and calculates a reference latency for each consonant byadding a positive value which is adapted to a duration or intensity of aconsonant portion to the reference latency for the vowel.
 11. Anannoyance judgment apparatus comprising: a presented-speech sounddetermination section configured to determine a monosyllabic speechsound to be presented by referring to a speech sound database retaininga plurality of monosyllabic speech sounds such that, for each speechsound, the speech sound and a reference latency of anelectroencephalogram negative component corresponding to the speechsound are retained in association; an annoyance judgment sectionconfigured to, in an electroencephalogram signal of a user measured by abiological signal measurement section, compare a peak latency of anegative component of the electroencephalogram signal in a range from 50ms to 350 ms from a starting point, the starting point being a point intime at which the speech sound is presented to the user by an outputsection, against the reference latency corresponding to the determinedspeech sound that is retained in the speech sound database, andoutputting a difference between the peak latency and the referencelatency; and an acoustic aiding processing section configured to adjustthe speech sound based on the difference output from the annoyancejudgment section.
 12. An annoyance judgment method comprising the stepsof: measuring an electroencephalogram signal of a user; determining amonosyllabic speech sound to be presented by referring to a speech sounddatabase retaining a plurality of monosyllabic speech sounds such that,for each speech sound, the speech sound and a reference latency of anelectroencephalogram negative component corresponding to the speechsound are retained in association; presenting the determined speechsound to the user; and judging annoyance of the output speech sound bycomparing a peak latency of a negative component of theelectroencephalogram signal in a range from 50 ms to 350 ms from astarting point, the starting point being a point in time at which thespeech sound is presented, against the reference latency correspondingto the determined speech sound that is retained in the speech sounddatabase.
 13. A computer program, stored on a non-transitorycomputer-readable medium, to be executed by a computer mounted in anannoyance judgment system for speech sound listening, wherein thecomputer program causes the computer in the annoyance judgment system toexecute the steps of: receiving an electroencephalogram signal of auser; determining a monosyllabic speech sound to be presented byreferring to a speech sound database retaining a plurality ofmonosyllabic speech sounds such that, for each speech sound, the speechsound and a reference latency of an electroencephalogram negativecomponent corresponding to the speech sound are retained in association;presenting the determined speech sound to the user; and judgingannoyance of the output speech sound by comparing a peak latency of anegative component of the electroencephalogram signal in a range from 50ms to 350 ms from a starting point, the starting point being a point intime at which the speech sound is presented, against the referencelatency corresponding to the determined speech sound that is retained inthe speech sound database.
 14. An annoyance judgment system comprising:a biological signal measurement section configured to measure anelectroencephalogram signal of a user; an audio input section configuredto input an audio signal of an utterance by a specified speaker; anaudio analysis section configured to output a trigger upon detecting atiming at which the audio signal is input, and analyzing acharacteristic feature of the audio concerning a duration and anintensity of a consonant portion; a reference latency estimation sectionconfigured to, based on the characteristic feature analyzed by the audioanalysis section, estimate a reference latency of a negative component;and an annoyance judgment section configured to judge annoyance bycomparing a peak latency of a negative component of theelectroencephalogram signal in a range from 50 ms to 350 ms from thetrigger as a starting point against the reference latency estimated bythe reference latency estimation section.
 15. The annoyance judgmentsystem of claim 14, further comprising a character output sectionconfigured to output text information indicating a speech sound for thespecified speaker to utter, wherein an audio signal of an utterance bythe specified speaker is input to the audio input section based on thetext information having been output from the character output section.16. The annoyance judgment system of claim 15, wherein, the characteroutput section further outputs information concerning a sound pressureindicating a loudness with which the specified speaker is to utter themonosyllabic speech sound; and an audio signal of an utterance by thespecified speaker is input to the audio input section based on the textinformation and information concerning sound pressure having been outputfrom the character output section.
 17. The annoyance judgment system ofclaim 15, further comprising a presented-speech sound determinationsection configured to determine a speech sound for the specified speakerto utter by referring to a previously-provided speech sound list,wherein the character output section outputs text information indicatingthe speech sound determined by the presented-speech sound determinationsection.
 18. The annoyance judgment system of claim 17, wherein thereference latency estimation section estimates the reference latency ofa negative component based on the characteristic feature analyzed by theaudio analysis section and on the speech sound for the specified speakerto utter that is determined by the presented-speech sound determinationsection.
 19. The annoyance judgment system of claim 17, wherein thereference latency estimation section estimates the reference latency ofa negative component by adding a predetermined positive value to apreviously-provided base latency, the predetermined positive value beingadapted to a consonant duration or consonant intensity of the audio. 20.An annoyance judgment method comprising the steps of: inputting an audiosignal of an utterance by a specified speaker; outputting a trigger upondetecting a timing at which the audio signal is input, and analyzing acharacteristic feature of the audio concerning a duration and anintensity of a consonant portion; estimating a reference latency of anegative component, based on the characteristic feature analyzed by theanalyzing step; and judging annoyance by comparing a peak latency of anegative component of the electroencephalogram signal in a range from 50ms to 350 ms from the trigger as a starting point against the referencelatency estimated by the estimating step.